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Introduction to This Guide 



What difference are we making ? 
How do we know? 

How can we show it? 



These questions about program performance are asked by adult literacy and 
ESOL teachers, program administrators, and funders. This guide is designed 
to be used by local adult education programs to facilitate a systematic inquiry 
process answering these kinds of questions. In this process, program staff take 
part in activities that involve them in identifying and clarifying program goals, 
examining current documentation processes, addressing the challenges of 
performance accountability and outcomes documentation at the program level. 
They produce a number of documents that their program can use (a) to make a 
decision about implementing ongoing improvement work and (b) to conduct 
this ongoing work. 

This inquiry guide was developed as part of a larger project of the National 
Center for the Study of Adult Learning and Literacy (NCSALL) that 
examined outcomes of adult literacy programs. The project included research 
on how adult learners identify these outcomes (Bingman and Ebert, 2000; 
Bingman, Ebert, and Smith, 1999), policy papers addressing performance 
accountability (Merrifield, 1998) and previous outcomes studies (Beder, 1999), 
and an action research project in which teams from three adult education 
programs in Tennessee, Virginia, and Kentucky examined their programs’ 
outcomes documentation processes and developed new processes designed to 
meet specific program needs (Bingman, 2001). 

Practitioners from three Tennessee programs field tested the inquiry process 
described in this guide and the guide was revised based on their suggestions. 

All the programs that took part in the field test made plans to continue 
program improvement work. 
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INTRODUCTION 



The program staff that participate in this inquiry process will have a clearer 
understanding of: 

• their goals, their program’s goals, and state and national goals 

• how to document the accomplishment of goals 

• performance accountability and what it means 

• how the various parts of their program fit together to move toward their goals 

• possible next steps their program can take to move more effectively toward 
their goals. 

When a program undertakes this inquiry process, one program staff member 
takes the responsibility of using the guide to facilitate a team of co-workers in 
six hour-and-a-half sessions. The activities in the sessions include: 

• examining current program work — goals, program activities, documentation 
processes 

• reading and discussing articles on performance accountability 

• examining state and national plans for the fits (and misfits) with program goals 

• beginning to develop possible next steps for program change. 

The guide includes: 

• an introduction for the facilitator 

• a step-by-step guide to facilitating the six sessions 

• reproducible materials for participants. 



Overview of the Six Sessions 



This inquiry process is divided into six sessions. The objectives and activities of 
each session are summarized below. 

Session 1: Examining Our Goals 

The objectives of Session 1 are for the group: 

• to get to know each other and the process 

• to examine and clarify student, staff, program, state and federal goals. 

The activities are designed to introduce the process, to get the group organized, 
to identify and clarify the goals of the staff, students, and program, and to 
compare local goals with state and national goals for adult education. 
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INTRODUCTION 



Session 2: The Documentation Matrix 

The objective of Session 2 is for the group: 

• to develop a clearer understanding of their program’s current 
documentation — what is documented and why? 

The activities include looking at the various documentation forms used by the 
program, completing a Documentation Matrix to analyze how they are used, and 
determining if there are program goals that are not documented. 

Session 3: Performance Accountability 

The objectives of Session 3 are for the group: 

• to have a greater understanding of performance accountability in adult 
education 

• to consider how and to whom their program is accountable. 

The activities for Session 3 include discussion of Juliet Merrifield’s article on 
performance accountability, assessing the local program on Merrifield’s criteria, 
and completing the Accountability Grid to analyze performance accountability 
in their program. 

Session 4: Inputs-to-Impacts 

The objectives for Session 4 are for the group: 

• to prioritize their program goals 

• to analyze how the components of their program address these program goals. 

The activities for Session 4 include prioritizing the goals identified in Session 
1, reviewing the components of the program using the Inputs-to-Impacts Model, 
and identifying any gaps in addressing program goals. 

Session 5: Documenting Outcomes, Measuring Performance 

The objectives for Session 5 are for the group: 

• to better understand outcomes documentation 

• to determine the outcomes that might be documented to assess program 
goals. 

The activities for Session 5 include discussing Harold Beder’s review of 
outcomes studies, identifying the outcomes that need to be documented in 
order to assess performance in meeting program goals, and examining several 
approaches to outcomes documentation. 
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INTRODUCTION 



Session 6: Considering Next Steps 

The objectives for Session 6 are for the group: 

• to understand three possible approaches to on-going program improvement 

• to determine next steps the program might take regarding program 
improvement. 

The activities for Session 6 include reporting on and discussing readings on 
approaches to program improvement, identifying gaps in current program 
processes, and identifying possible next steps the program might take toward 
continuous improvement. 



To Implement an Inquiry Group 



An adult education program interested in exploring or beginning a program 

improvement process using this inquiry process should take the following steps: 

1 . Decide that the process will be used as a way to begin to examine the results 
of your work. 

2. Choose a team of four to twelve staff members, teachers, and administrators, 
who are willing to spend a couple of hours a week for six weeks on this 
process. 

3. Identify a facilitator who will take the responsibility of reading the guide 
and leading the activities. This might be a shared responsibility.* 

4. Schedule six sessions. 

5. Begin! 



*“Getting Started” (pp. 6-7) has suggestions on choosing the team and facilitator. 
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Introduction for the Facilitator 



This section is written for the person chosen by the program to facilitate the 
inquiry process. You will find an introduction to your role, suggestions on how 
to organize the process, and steps to take before the first session. Suggestions on 
facilitation skills are found in Appendix 2. 



Introduction to the Process and Your Role 



This guide is designed to assist adult education programs in using an inquiry 
process to develop information about their program in order to plan for 
program improvement. The team that takes part in the inquiry will spend 
approximately an hour and a half a week together working, talking, and 
planning. Each team member will have some reading and other activities to 
prepare for the sessions. Each team member will complete some documents 
that will be used throughout the process. 

As facilitator, you will serve as a guide to the inquiry process. Your role will 
include: 

• helping the group get organized and stay organized 

• leading the group through each session 

• keeping the process moving. 

In this inquiry process you are the leader of a team of equals. This process is 
not about giving out information or new policies. It is about a group of people 
working together to build their understanding and to develop new information 
and, possibly, new policies. You may be a program administrator or a teacher, 
but as facilitator of this inquiry process, your primary role is that of guide. A 
facilitator makes things happen but does not dictate. You are a part of the group 
in this inquiry process, but you have extra responsibilities for the group’s success. 
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FACILITATOR’S INTRODUCTION 



Organizing the Inquiry Process 



This section of the Guide describes the initial steps the facilitator should take 
to organize an inquiry group. 



Getting Started 

How the decision is made to conduct this inquiry process in a program will vary. 
Maybe a program administrator has learned about the process and decided to try 
it. Maybe a group of teachers heard a presentation and think the process will be 
useful. Maybe the state has suggested that programs use this process to develop 
an improvement plan. Whatever the reason, once a program decides to conduct 
an inquiry into program improvement, two other decisions need to be made: 

1. Who will facilitate the process? 

2. Who will participate in the process? 

The facilitator of the inquiry process is both a participant and a leader of the 
process. In some group processes the facilitator is “outside” the process. In this 
inquiry process, the facilitator guides the group through the process, but may 
also make his or her own contributions. The ideal facilitator will have some 
experience in group facilitation. The facilitator will need to be able to commit 
the extra time needed to prepare for each session, approximately four hours 
before the first session and an hour or two for each of the other sessions. 

The decision about who will participate in the process may be made by program 
administration, by the facilitator, or group members may volunteer. However the 
group membership is determined, it is important to pay attention to: 

• The number of group members. Four is the minimum and twelve the 
maximum. The best number for discussion is probably between six and ten. 

• The makeup of the group. The group that participates in the process 
should include a diversity of program staff, at a minimum, several teachers 
and at least one program administrator. If this inquiry process is to lead to 
program change, the work and the new understandings that the work leads to 
will be stronger if various viewpoints are represented. For program change to 
happen, broad'based understanding and commitment is vital. As you think 
about the group, consider the various viewpoints that are represented in your 
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FACILITATOR’S INTRODUCTION 



program; for example, you may want to include ESOL teachers and ABE 
teachers if your program offers both, or you may include a volunteer tutor if 
volunteers are an important part of your program. 

• The commitment of the group. It is important that group members are 
committed to the process. They need to make every effort to attend each 
session and to be prepared. The schedule of sessions should be set with group 
input. In the first session, ground rules are established to clarify each person’s 
contribution to the process. The program may want to consider some kind of 
stipend or honorarium for this extra commitment, particularly for part-time 
staff and volunteers. 

In the Participants Materials in Appendix 1 is an Introductory Letter that may 
be used or adapted to recruit group members. 



Preparation 

Once the group members have been identified, the facilitator needs to do 
several things to prepare for the inquiry group. 

First, be sure the schedule of session meetings has been set or that at least the 
first meeting has been set with plans to set the rest of the schedule with the 
group. Although each session should take 1.5 hours, ask participants to set 
aside two hours, so that you will not have to cut short an interesting discussion 
or activity. A Schedule/To Do Form is included in the Participants Materials, 
and you may want to complete it for the first session. 

Second, read through this guide. Get a sense of each session, scan the readings, 
get an understanding of the overall process. 

Third, brush up on your facilitation skills. If you are a novice facilitator, you 
may want to read all of the materials in Appendix 2. Even if you are a seasoned 
facilitator, you will probably want to review the materials in the appendix. The 
Guide is very specific about what to do in each session, but every group is 
different and the process is interactive. You will want to think ahead about the 
challenges you might face in the group, and prepare for any particular 
challenges you have as a facilitator. 
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FACILITATOR’S INTRODUCTION 



Fourth, you will need to decide if you want to include the optional state goals 
in Activity 4 in Session 1. If you do, you will need to obtain a copy of your 
state’s goals for adult basic education and ESOL. These goals (or mission) are 
listed in the state plan. See Appendix 3 for information on how to get your 
state plan and what to excerpt. 



Staying Organized 

This guide provides a variety of tools to help you and the group stay organized. 
You may want to keep your materials in a loose-leaf notebook or binder. You 
should provide each inquiry group participant with a folder for their 
Participants’ Materials. 

The Participants’ Materials section includes reproducible master copies of: 

• the Introductory Letter 

• the Overview 

• the Schedule/To Do Form 

• an optional Feedback Form for sessions 1-5 

• the handouts and readings for each session. 

Participants’ materials are italicized when referred to in the text. 

Documenting Your Work 

At the end of this project, each participant should have a record of your work. 
You may also want to present your work in a portfolio to your entire program 
staff at the end of the project. Most work will be done on newsprint. The 
completed newsprints are often used in more than one session. And the 
information on the newsprints should be copied onto the corresponding forms. 
You should decide if you want individual participants to complete their own 
forms or if you or a volunteer will type up your results and copy and distribute. 
One of the pilot teams also created a poster that helped document their 
ongoing work. However you choose to do it, each person should have copies of 
the completed work, and you should keep “master” copies of all documents 
produced by the group to be used in any future program planning and 
improvement work. 
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FACILITATOR’S INTRODUCTION 



Before the First Session 



Before the first session: 

1. Be certain you are clear about any administrative issues such as payment for 
group members, etc. 

2. Make a Participants’ List with name, address, phone, email, and position. 
Make copies for the group members and keep a copy for the master file. 

3. Distribute the Introductory Letter to group members if this hasn’t already 
been done. 

4- Obtain a copy of your state plan or other statement of your state’s goals for 
adult education (optional, see Appendix 3). 

5. Decide if you want to give folders of materials to each group member. There 
are several possible ways to do this: 

— Pass out an empty folder and let people put in materials as they receive 
them OR 

— Put all the handouts in a folder and give out at the first session OR 
— Give out all the handouts and readings before the sessions begin and 
suggest that participants review the agenda and the readings. 

6. Think about ways to make the process fun. Do you want to serve snacks 
either before or after the sessions? If so, is there a way to share the work? 



The Sessions 



The other six sections of the Guide are the facilitator’s directions for each 
session of the inquiry process. The session directions include the objectives 
of the session, the preparation needed, and a materials checklist. Then each 
activity in the session is described in detail, with purpose, estimated time, 
materials, and numbered steps in the process. 



You will probably want to participate in some activities, and this is fine. But 
remember, your first responsibility is facilitation. 
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SESSION 1: 

Examining Our Goals 

The objectives of Session 1 are for the participants: 

• to get to know each other and the process 

• to examine and clarify student, staff, program, and state and federal goals. 



Preparation 

This first session lays the groundwork for the rest of the process. The group will 
come together, begin to experience the process, and examine the goals that 
guide your program. You will want to be as prepared as possible. 

Before the session, gather the materials listed below. Make all the copies you 
need. In addition to the Participants’ List, you will need to create one handout, 
the excerpts from your state plan, if you have decided to include it. You will 
find instructions for this in Appendix 3. The other handouts are in Appendix 
1. In addition to handouts, you also need to create the newsprints as described 
in the Activities. 

After you have all the materials ready, read through the instructions again and 
highlight words that will remind you at a glance of what you need to do or say. 
Imagine how the activity will go and how long it should take. 
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SESSION 1 



Materials 

Newsprint pad 

Markers 

Masking tape 

3" x 5" Post-It notes or “stickies” in three colors 

Folders for each group member with Participants’ Materials List 

Handouts (copies for each group member, hole-punched): 

Participants’ List (facilitator creates) 

. Overview 

Schedule/To Do Form 

Goals Sheet 

Reading # 1 - Workforce Investment Act Introduction 

Excerpts from state plan (optional - facilitator creates) 

Feedback Forms OR feedback questions written on newsprint 

Newsprints prepared for: 

/iyeuda (Activity 1) 

Student tfoole, Sto^ tfooie, 'Proynom tfazte (Activity 3) 

Ptoytam tfaoie (Activity 3, optional) 

'pedenol tfade and State tfade (Activity 4) 

Feedback questions (Activity 5, optional) 
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SESSION 1 



Activity 1. Introduction 

Purpose To familiarize the group with how the inquiry process will work 

Time 15 minutes 

Materials — Overview handout 

— Agenda 

— written on newsprint 

Process: 

1. Welcome the Group 

Introduce yourself and state your role as facilitator. Explain that your role 
includes 

• providing needed materials 

• leading the group through each session 

• keeping the process moving. 

2. Ask each person to introduce themselves and tell their position in the 
program. 

3. Hand out Agenda and post the agenda for the day on newsprint. 

Review briefly and ask if anyone has any questions. Explain that they can 
use their agenda and the space for reflection to keep a record of their own 
experiences in this inquiry process. 

4. Introduce the process 

Explain that for the next few weeks they will 
be part of an inquiry group that will be looking 
at how your program determines how well you 
are meeting your goals. The process will help 
them to: 

• identify or clarify program goals 

• examine current documentation processes 

• begin to identify possible ways your program 
can better meet your goals. 

At the end of the six sessions, the group may 
decide to continue to meet and work together, 
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Agenda 

1 . Introduction — to people and 
the process. 

2. Getting organized — schedule, 
ground rules, keeping it together. 

3. Review of our goals. 

4. Looking at state and national 
goals. 

5. Feedback and getting ready for 
the next session. 
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but their commitment is to participate in the six sessions and present their 
work to the program. Explain that the work they do in the sessions will 
result in documents they will give to the program. They do not have to 
write a report. 

5. Overview of the Process 

• Pass out the Overview (or ask people to find it in their folder if they 
already have it). 

• Ask people to refer to the Overview while you review it. 

• Explain that the work of the group will include: 

— clarifying your goals 

— examining what your program currently documents and how you use 
this documentation 

— reading and discussing an article on performance accountability 

— examining how the components of your program interact 

— exploring a variety of ways to document outcomes 

— exploring some possible ways to continue to work on program 
improvement. 

Ask the group to look over the Overview. Ask if there are questions about 
the process; answer based on your understanding of the process. If there are 
questions that are not about the process, note them and say you will try to 
get the answer, but do not get involved in answering now. 



Activity 2. Getting Organized 

Purpose: To set ground rules, schedule, materials 

Time: 15 minutes 

Materials: — newsprint 

— Schedule/To Do Form 
— calendar 
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Process 

1. Set Ground Rules 

Ask the group to think a minute about ground rules for their work together. 
After a moment ask for suggestions and write on newsprint. If people need 
help, some suggestions might include “Avoid interrupting” or “Share air 
time.” After all the suggestions are posted, ask if everyone can live with 
these. Tell the group you will post these at subsequent sessions as a reminder 
and that additional rules can be posted as needed. (Save newsprint.) 

2. Clarify Schedule 

If the schedule is already set, fill in the schedule form ahead of time and 
pass out. Ask if anyone has problems with this schedule. If one or two 
people must miss one or two sessions, this is probably okay, but if several 
people must miss a session, discuss re-scheduling. 

If the schedule is not already set propose the dates you think are most likely. 
Try to come to consensus as quickly as possible. 



Activity 3. Examine Goals for Adult Education 

Purpose To clarify the goals of your program 

Time 30 minutes 

Materials — Goals Sheet — a copy for each participant 

— Newprints with Sta^ (fails, Student (fails . , (fails. 

— “Stickies” (3”x 5” post-it notes) in three colors 
— Newsprint, markers, tape 

— If your program has established written “official” program 
goals, copy these on a sheet of newsprint. 

Process 

1. Writing Goals 

Explain that the purpose of this activity is to clarify the goals that influence 
your program. These include staff goals, student goals, and program goals. 
The assumption is that goals should drive the activities of a program. 
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Post three pieces of newsprint, one labeled Stu^ tfavU, one Student tfasvU, 
and one Pn^nunt. tyoafo. Put a stickie of a different color in the upper right 
corner of each. 

Give each person five stickies of each color, a total of 15. Ask them to think 
a few moments of their own goals for their work, what they hope they are 
accomplishing by teaching in or administering an adult education program. 
Ask them to write their goals, using the stickies of the Staff color. They 
should put one goal per stickie, up to 5. Allow about 5 minutes. 

Then ask them to think about the students in their program and the goals 
they know or believe are most important to their students. They should 
write one student goal per stickie on the Student color. 

Finally, ask the group to write what they believe are their program’s goals, 
again one per stickie, on the Program color. As they finish, each person 
should take their stickies and post them on the appropriate newsprint. 

2. Sorting the Goals 

Ask for two volunteers to come up and cluster the goals on each sheet, i.e. 
two people for each sheet. They should group goals that are similar. 

When they are done, ask them to report on their grouping and to write a 
phrase for each grouping. You should end up with a list of goals for each 
sheet. (Save these goal lists.) 

Ask for thoughts on what they have done: 

What are the similarities? The differences? Any surprises? 

3. If your program has established “official” goals, put up a newsprint 
page on which you have written them. Ask the group if the official 
goals adequately reflect the lists they have developed. If there are important 
differences, what are they? Note the difference on the pfioytam. (faula. sheet 
or the Official tfavU. 

4. Tell the group that you will come back to these lists in another 
session. Depending on time, they can copy the final lists on their Goals 
Sheet or you (or a volunteer from the group) can copy them on the Goals 
Sheet later and make copies for each group member. 
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Activity 4. Review State and Federal Goals 

Purpose To examine state and national goals for adult education and 
compare to local goals 

Time 20 minutes 

Materials — Newsprint lists of Sta££, Stud&tt, and from 

Activity 3 

— Reading # 1 - Workforce Investment Act Introduction 
— Handout - Excerpt from your state plan (optional) 

— Newsprints labeled 'ped&uzt (fazle and State (fazle 
(optional) 

Process 

1. Examine state and federal goals. (If you have decided not to include the 
state plan, just focus on federal goals from WIA.) 

Pass out Reading #1 and state plan excerpt. Tell the group that they may 
want to read the entire handout later, but for now they should scan to find 
the goals stated in each. 

Ask someone to read the state goals. Write on newsprint. Repeat for federal 
WIA goals. 

2. Discuss 

Facilitate a group discussion asking the following questions: 

• How do these goals compare with local goals? 

• How do your program goals align with state and federal goals? 

• Are there others you would add to the state or federal lists? 

Summarize the group’s responses in a few notes on the state and national 
goals newsprints. Tell them that you will be returning to these goal sheets in 
another session. 
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Activity 5. Wrap-Up 

Purpose To get feedback on the day’s session and to prepare for next 
session. 

Time 10 minutes 

Materials — Folders (if not already distributed) 

— Feedback Forms OR feedback questions written on newsprint 

Process 

1 . Tell the group that in the next meeting they will examine the 
documentation that your program is now doing and how it is used. 

Ask the teachers to bring any form, test, portfolio, etc. that they use 
regularly. This might include intake forms, goal sheets, tests (standardized 
and other). Ask administrative staff to bring program forms — the forms they 
regularly complete and keep or forward to another office. Explain that these 
forms will be returned at the end of the session. 

2. Pass out folders (if not already done). 

Ask the group to look at the Participants’ Materials List. These are the items 
they will eventually have in their folders. They should organize them how 
they like. 

3. Complete a quick feedback process. 

Either pass out the Feedback Forms and 
ask people to take about 5 minutes to 
complete and return to you OR post the 
newsprint with the Feedback questions 
and ask for oral responses that you note 
on the newsprint. 



What questions do you have 
from this session? 



I want the facilitator to know 



I think I can use what I 
learned today to 
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SESSION 2: 

The Documentation Matrix 



The objective of Session 2 is for the group: 

• to develop a clearer understanding of their program’s current documentation: 
what is documented and why? 



Preparation 

This is a working session. You will need table space to sort documents and a 
large black, white, or green board and the appropriate writing tools. If you do 
not have access to a board, create a large paper by taping together newsprint. 
Then on the board or paper recreate the Documentation Matrix. 

The other preparation for this activity is to encourage group members to bring 
all the kinds of documentation used in your classes and program. 
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Materials 

A “board” and chalk or markers 

Newsprint pad, markers, tape 

Newsprint sheets summarizing goals from Session 1 , Activities 3 & 4 

Handouts (copies for each group member, hole-punched) 

Agenda 

Documentation Matrix (2 copies per person and have extras available) 

Feedback Forms 

Reading # 2 - Performance Accountability: For What? To Whom? How? 
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Activity 1 . Check In 

Purpose To reconnect with the inquiry process and review agenda 
Time 10 minutes 

Materials Agenda 
Process 

1. Welcome the group. 

2. Ask if there are any concerns or questions from the last meeting. 

Answer or discuss briefly, asking for input from the group. 

3. Ask if anyone has any thoughts or insights about goals since the last 
meeting and if they’d share. Acknowledge these contributions. 

4. Pass out and review the day’s agenda, i.e. that the rest of the session 
will be spent sorting and analyzing the documents they have brought. 



Activity 2. Sorting the Documents 

Purpose To identify and categorize the kinds of documents the group has 
brought 

Time 15 minutes 

Materials — newsprint and markers 

— the documents that group members brought 

Process 

1. Explain to the group that the activity of the day is to make sense of 
the documentation that your program uses. In the last session you 
identified goals; by the end of the day you should have a sense of how you 
are keeping track of your work and success in meeting these goals. 
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2. Ask the group what kinds of documents they have brought. You may 

want to ask each person to name one kind of document, e.g. a standardized 
test reporting form or a student journal, as a way to get everyone’s 
participation. As people name the type or category write it on newsprint. 
Keep going around the group until you have elicited all the categories. 

Ask the group to look at the list and note if there are any categories that 
might be combined. Connect and write title for combined categories on 
newsprint list. 

3. Ask the group to take the documents they brought and physically 
sort them using the categories they have developed. If there are any that 
don’t fit, create a new category, even if it only has one document in it. 
When all the documents are sorted, ask everyone to sit again taking one 
or more category piles with them. 



Activity 3. Completing the Matrix 

Purpose To analyze your program’s documentation processes 

Time 45 minutes 

Materials — Documentation Matrix handouts 

— An enlarged 'TftsiPUx on a board or paper 

Process 

1 . Explain that you are now going to work together to analyze your 
documentation processes using the questions on the matrix. 

First write the categories you have identified in the first column of the 
enlarged 'TftaPUx under Documentation Method, one category per row. (You 
may want to start with the document that is completed first for a student 
and list them in order, but this may not work consistently.) 

2. Then go through the list of documentation methods, item by item and 
ask the questions at the top of the columns. When the group agrees on an 
answer, write it in the appropriate place. People may want to refer to the 
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various piles of documents as they answer. And you may decide to combine 
additional categories. A sample matrix is shown below, for your information. 



Documentation 

Method 


Who does it? 


For whom? 


How often? 


How used? 


Key 

Information 


Intake Form 


secretary 


program 


once 


for demographic 
info to set goals 


• demographics 

• goals 

• previous 
education 


TABE test 


teacher 


• student 

• teacher 

• state 


every 
90 days 


• to assess 
progress 

• for NRS report 


grade level 


Attendance 

Form 


teacher 


program 


every day 


to report to 
state 


attendance 



The questions mean: 

Who does it? means who actually completes the form. 

For whom? means who ultimately uses the information on the form. This 
may be more than one person. 

How often? means how often is this information collected per student. So 
each student usually has one intake form even though the program may 
complete many. 

How used? means how is the information used. This may be for 
performance accountability, to plan lessons, or for a variety of purposes. 
Key items reported refers to the types of information not to specific 
information. 

3. After the is completed either give time for people to copy 

onto their forms or you or a volunteer copy the matrix onto a form after 
the meeting and make copies for everyone. 
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Activity 4. Which Goals Are Documented? 



Purpose To determine which of your program’s goals are documented 
Time 15 minutes 

Materials — The enlarged completed ’TJtatmx on a board or paper 
— (fail*. Aiafo from Session 1 (newsprint) 

Process 

1. Post the (fad& (student, staff, program, state, national) from Session 1. 
Ask the group to work in pairs to identify for which goals your program 
collects documentation of achievement. For which goals do you collect data 
that tells you when your goals are met? 

2. Ask pairs to report which goals are documented and in what 
document. Go around the group asking each pair to report one goal and 
how it is documented. Then ask if there are others. On the (fad Aiato, 
star (★) the goals for which you document achievement. (You may want to 
number the types of documents in column 1 of the ‘Tt&sttmx and write the 
number next to the goal it documents.) 

3. Ask the group for comments, reflections on what they have done both 
on the ‘Tt&stftex and comparing it with goals. 



Activity 5. Wrap-up 



Purpose 



To give reminders for next session and get feedback from this 
session. 



Time 10 minutes 



Materials — Feedback Forms 

— Copies of Reading #2 - Performance Accountability: For What ? 
To Whom ? How ? 
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Process 

1. Pass out Feedback Forms . Give time to complete and collect. 

2. Remind people of the date of the next session. Ask them to read Juliet 
Merrifield’s article, “Performance Accountability: For What ? To Whom ? How?” 
and to write one question the article raised for them. 

Tell them that although this article was written before the passage of the 
Workforce Investment Act and the implementation of the National 
Reporting System, it is still useful for understanding the underlying issues 
in performance accountability. 

3. Tell them that if they want to read the report on which the article is 
based it is available ondine at 

http://gseweb.harvard.edu/~ncsall/research/reports.htm 
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Performance Accountability 

The objectives of Session 3 are for the group: 

• to have a greater understanding of performance accountability in adult 
education 

• to consider how and to whom their program is accountable. 

Preparation 

To prepare for this session, read Merrifield’s article and consider your own 
answers to the discussion questions in Activity 1. When facilitating a 
discussion your role is to pose questions and in some cases asking clarifying 
questions. It is not to get the group to give you a “correct” answer. The purpose 
of the discussion is to deepen the group’s understanding of performance 
accountability and what it does and could mean for your program. However, 
considering how you might answer the questions should give you a deeper 
understanding of the article. 
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Materials 

Newsprint 

Markers, tape 

Handouts (copies for each group member, hole-punched) 

Agenda 

Accountability Grid 

Feedback Forms 

Newsprints 

Newsprints labeled: 

s4&cou*tt<z&te 0 ?<vi / rfeaucttfaMe "7a "ZO&twi, / /4ccau*ita&le "pon, “TM&at 
(Activity 2) 

2uote& from article (Activity 3) 

‘Tlext Stefa newsprint (Activity 4) 
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Activity 1 . Check In 



Purpose To reconnect with the inquiry process and review agenda 
Time 5 minutes 

Materials Agenda 
Process 

1. Welcome the group. Ask if anyone has comments or thoughts to share. 

2. Pass out and review the day’s agenda: completing an Accountability 
Grid to clarify your program’s accountability system, discussing the article 
they have read, and using Merrifield’s ideas to consider performance 
accountability in your program. 



Activity 2. The Accountability Grid 



Purpose To consider how accountability happens in your program 

Time 30 minutes 

Materials — Newsprints labeled: /l&&atwta&le. pan / /IcMtwttdle, Pa Tt/tunn / 
/tccatwfaMe pan. ’W&at 

— Markers 

— Accountability Grids - 2 copies for each person 

Process 

1. Pass out the Accountability Qrids and explain that they will begin by 
thinking about how they are accountable in their work. 

2. Ask participants to think a moment, then write a list of what they are 
accountable for in their work in the Accountable For column of their grid. 

3. Go around the group and ask each person to tell two things on their 
list. Write each on a sheet of newsprint labeled /tcaxMtti&le, pan.. After every 
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person has spoken go around again and ask for any items not already on the 
list and record those. 

4. Now ask the group to think to whom are they accountable. Using 
the same process as above have them create their own lists. Then list the 
various types of people or organizations named, e.g. students, program 
director, state ABE, etc. on another sheet of newsprint labeled 

5. Ask the group to think a moment about the outcomes they measure 
and report and how. Remind them that outcomes are what comes out of 
the program, usually changes in learners’ lives; not what goes in, such as 
instruction. Ask them to list the ways they measure and report outcomes on 
the grid column labeled Accountable How. For example, they might use a 
standardized test to measure how learners’ reading has changed. 

6. Using the same process as above, go around the group and ask each 

person to share two outcomes they measure and how. List these on a 
sheet of newsprint labeled go around again and ask for any 

not yet reported and add these to the list. Tell the group that these lists are 
a beginning picture of performance accountability in their programs. 



Activity 3. Merrifieid on Performance Accountability 



Purpose 


To discuss Merrifield’s article and deepen the group’s 
understanding of performance accountability 


Time 


30 minutes 


Materials 


Quotes written on newsprint 
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Process 

1. Ask for the group’s reaction to the article. 

2. After everyone has had a chance to comment, post a newsprint with 
the following quotes: 

tZccotea,: 

“[With WIA] The emphasis on results shifts attention from simple delivery of 
services to the outcomes of learning.” 

“Definitions of success should be negotiated.” 

“Two kinds of capacity — to perform and to be accountable — are linked.” 
“What is counted becomes what counts.” 

“Inequalities of power and uneven access to information prevent the 
development of mutual accountability.” 



Facilitate a discussion of Merrifield’s article using the following questions 
for the quotes. Some possible answers are in brackets in case you need to 
suggest answers. Your objective is to help the group be clear about: 

• agreement on performance 

• capacity to be accountable 

• tools for measuring accountability 

• mutual accountability. 

You may not want to ask all the questions if time is short. 

Quotes, discussion questions, and possible answers: 

“[With WIA] The emphasis on results shifts attention from simple delivery of 
services to the outcomes of learning.” 

— What services does our program provide? [instruction, counseling, testing] 
— How are services different from outcomes? [Services are what you do. 
Outcomes are the results you hope for.] 

“Definitions of success should be negotiated.” 

— Who determines what success means in your program? 

— Do definitions of success differ? 

— How might a program-wide definition be determined? Who should be 
involved? 
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“Two kinds of capacity — to perform and to be accountable — are linked.” 

— How are these two capacities different? [Capacity to perform involves 
having the resources to meet your goals - time, trained staff, etc. Capacity 
to be accountable involves the collecting and reporting of data that 
measures performance, e.g., reliable testing, accurate reporting.] 

“What is counted becomes what counts.” 

— Think about the Documentation Matrix. What is counted in your program? 

— Do you see ways that what is counted affects how you do your work? 

— Do people “teach to the test” and if so, is that a problem? 

“Inequalities of power and uneven access to information prevent the 

development of mutual accountability.” 

— What does Merrifield mean by “mutual accountability?” [All stakeholders 
in adult education have access to information and responsibilities to each 
other.] 

— What might get in the way of mutual accountability? 



Activity 4. Planning Next Steps 

Purpose To begin planning using Merrifield’s “next steps” 

Time 20 minutes 

Materials — Hext Stefa newsprint 

— 3" x 5" stickies - any color 

Process 

1. Tell the group that while some aspects of performance accountability 
are currently mandated, as a program, you also have opportunities to use 
performance accountability for program improvement. This activity focuses 
on ways you might do that. 
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2. Post the 'Hext Stefa newsprint. Leave space to write between each item. 
While you are writing ask a group member to give everyone at least 5 stickies. 

'Hext Stefa: 

— Agree on performance. 

— Build capacity to perform. 

— Build capacity to be accountable. 

— Develop and use tools to measure the performance that you value. 

— Develop mutual accountability. 

3. Ask the group to think silently of at least one step your program could 
take toward each of these. They should write one suggestion per step on a 
stickie and after they are done put their stickies under the appropriate step. 

4. Look at the suggestions and group quickly. Read the categories of 
suggestions for each step and ask if there are others. Tell the group you will 
reproduce this into a combined list and give them a copy at the next 
meeting. 



Activity 5. Wrap-Up 



Purpose To prepare for next session and get feedback on Session 3. 

Time 5 minutes 

Materials — Completed Goals Sheets handouts from Session 1 

— Completed Documentation Matrix handouts from Session 2 
— Feedback Forms 

Process 

1 . Hand out the complete Qoals Sheets and Documentation Matrix OR 
refer them to the forms they have completed. Tell the group that at 
the next session they will be prioritizing program goals. Ask them to review 
the Goals Sheets with students and other program staff. 

2. Pass out Feedback Forms. Collect. 

Ask for any final questions or comments. Close. 
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InputS'tO'Impacts 

The objectives for Session 4 are for the group: 

• to prioritize program goals 

• to analyze how the components of your program address program goals. 

Preparation 

This session has two main activities, prioritizing program goals and completing 
an InputS'tO'Impacts model. Most programs have official goals, but these may 
or may not include everyone’s goals. The goabprioritizing activity returns to the 
Goals Sheets completed in Session 1, and the group prioritizes these goals. Then 
based on these program goals, the group completes the InputS'tO'Impacts Model 
and uses the form to consider how your program is addressing your goals. 

To prepare for this session, you will need to create an expanded version of the 
InputS'tO'Impacts Model using a white board or several pieces of paper. You may 
want to create two long forms, one for student factors and one for program 
factors to use in Activity 3. 
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Materials 

Newsprint and markers 

Sticky dots 

Newsprint goals sheets from Session 1 

Inputs-to-lmpacts form on large paper 

Handouts (copies for each group member, hole-punched) 

Agenda 

Glossary 

Inputs-to-lmpacts Model 

Sample Inputs-to-lmpacts Model 

Reading #3 - Lessons from NCS ALL’S Outcomes and Impacts Study 

Worksheet for Lessons article 

Feedback Forms 
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Activity 1 . Check In 



Purpose 


To reconnect with the inquiry process and review agenda 


Time 


5 minutes 




Materials 


Agenda 




Process 

1. Welcome the group. Ask if anyone has comments or 


thoughts to share. 


2. Pass out and review the day’s agenda. 




Activity 2. 


Prioritizing Program Goals 




Purpose 


To determine which of many possible program goals are priorities 
for this group 


Time 


25 minutes 




Materials 


— N ewsprint and markers 

— Sticky dots, in three colors 

— Aiafo on newsprint from Session 1 





Process 

1 . Explain that today you are going to prioritize the goals that were 
listed in Session 1. Say that while your program may have set “official” 
goals, and that the state and federal systems have established goals, your lists 
contain the goals of many stakeholders in your program. Today’s priority- 
setting process will establish a set of goals for your group. 

2. Give each person seven dots of one color. Ask them to look over the 
goals on the three lists: staff, student, program. Ask them to think about any 
conversations they may have had since Session 1 about goals. Then place 
their dots on the seven goals they believe are the top priority for your 
program. 
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3. When the first round of “voting” is completed give each person 

3 dots of another color and ask them to vote for their top three goals from 
the commonly chosen goals. When everyone has voted, you will have a list 
of goals that your group holds in common with indication of those that have 
highest priority for your group. Make a newsprint list of the P'Umittf tfaafo 
that have more than one vote. Save. 



Activity 3. Review Inputs-to-lmpacts Model 



Purpose To understand the items on the model 

Time 5 minutes 

Materials — Glossary 

— InputS'tO'Impacts Model 

Process 

1. Hand out the Qlossary and blank Inputs-to-lmpacts Model . Explain that 
the Inputs-to-lmpacts Model is a logic model developed to clarify the 
various aspects of adult education program processes. Student and program 
factors are listed in separate columns. This Inputs-to-lmpacts framework can 
be an important tool for analyzing broad program structure and how 
particular activities support program goals. 

2. Review terms in Qlossary . 

In the model, Input is defined as what the student or program brings to 
the processes of the program. Processes are the educational and 
organizational activities that take place in the program. Outputs is 
defined as the immediate results of these processes, while outcomes are 
the changes that occur in students’ lives as a result of their participation 
in the program, or the long-term results of program improvement. 

Impacts are the changes in the community resulting from changes in 
students and programs. 
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Activity 4. Complete Model for Program 

Purpose To examine how program components connect 

Time 40 minutes 

Materials — InputS'toAmpacts Model 

— Sample InputS'tO'Impacts Model 
— Expanded 'Impacto "TKadel on newsprint 

Process 

1. For this activity, divide the group into two teams. One team will work 
on the Student column and one will work on the Program column. Ask 
each team to decide what should go in each box in their column of the grid. 
For the outcomes and impacts rows, the items will be what they believe are 
the outcomes and impacts of your program. They should begin by having 
each person complete his or her own form and then combine. When they 
have reached agreement, they should fill in the expanded version. 

2. When each team is through, they should read the other team’s work 
and add items as they see the need. 

3. Pass out sample forms and review. Ask if there are any changes people 
want to make on the completed forms. If so, discuss, and add if there is 
agreement. 



Activity 5. Review the Model 



Purpose To review the model in light of goals 
Time 15 minutes 

Materials — Completed “WUdel on newsprint 

— (faifo Aiat (on newsprint) from Activity 2 

Process 

1 . Ask the group to look again at their priority goals and to look at the 

‘Tttodei. Discuss the following questions for a few moments: 
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• Are the processes needed to meet your goals in place? 

• Do you need additional inputs? 

2. After a brief discussion ask each person to make notes on their 
blank Inputs-to-Impacts Model about changes they believe might better 
enable your program to meet the goals you have prioritized. Ask them to 
save these forms to use in Session 6. 



Activity 6. Wrap-Up 

Purpose To prepare for next session and check-in 

Time 5 minutes 

Materials — Harold Beder’s article - Reading #3 

— Worksheet for article 
— Feedback Form 

Process 

1. Tell the group that in the next session you will discuss outcomes 
measurement and documentation and give some examples from ABE 
and other fields. 

2. Pass out Harold Beder’s article and worksheet. Tell the group that this 
article examines some national outcomes studies. Because the article is long, 
some of the article should be divided among the group. Everyone should 
read the first two pages and from “The Answer?” on page 7 to the end. 
Divide the four studies (1973 National Evaluation, page 2; California GAIN 
Study, page 3; National Evenstart Evaluation, page 4; the NEAEP, page 5) 
among the participants in the group, making sure that at least one person 
reads each. Tell them they will be asked to report on what that study found 
in the next session. Suggest they note the study they are going to read on 
the worksheet next to question 6. The worksheet is designed to connect the 
article to thinking about your local program. Ask the group to look it over 
and complete it as they read the article. 

3. Pass out and collect Feedback Forms . 
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Documenting Outcomes, 
Measuring Performance 



The objectives for Session 5 are for the group: 

• to examine different approaches to documenting outcomes 

• to begin to identify ways to document the outcomes that your program wants. 

Preparation 

This session begins with a review of work to date and a brief introduction to 
outcomes measurement. The activities include a review of Beder’s article on 
large-scale studies, an activity to explore quality-of-life measurement, and an 
activity using a locally developed outcomes documentation tool. In the final 
activity, the group returns to the Documentation Matrix and their P^^ncty (fasdt 
and, drawing on what they have learned from the other activities in Session 5, 
begin to identify documentation processes for your program’s goals — ways to 
know “how we are doing.” 

To prepare for this session, you will need to have the newsprint-size Pruvuty 
tyoafo list created in Session 4 and the large T><yccmtettteUOut ‘Tttatmx from Session 
2. Participants should also have these as handouts. They will also need the 
Workforce Investment Act Reading from Session 1. You may want to have a few 
extra copies available. You will need to make copies of the Quality' of 'Life 
Measurement; the DoSetMet Goals Sheet; Reading #4 - the article by Cody, 

Ford, and Hayward; and create newsprints for Activities 2 and 4. 
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Materials 

Newsprint and markers 

tyxzfo Ai&t from Session 4 

Large Ttta&Ux from Session 2 



Handouts 

Agenda 

Quality-of-Life Measurement 

DoSetMet Goals Sheet 

Reading #4 - “A Story of Improvement” 



Newsprints 

Newsprint with review of the work done so far - Activity 2 

Aiot 'JocUco&vu (on newsprint) - Activity 4 
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Activity 1 . Check In and Agenda Review 

Purpose To reconnect with the inquiry process and review agenda 
Time 5 minutes 

Materials Agenda 
Process 

1. Welcome the group. Ask if anyone has comments or thoughts to share. 

2. Pass out and review the day’s agenda. 



Activity 2. Introduction to Outcomes Measurement 



Purpose To link the concept of outcomes to the work done so far 

Time 5 minutes 

Materials Newsprint and markers 

Process 

1 . Tell the group that in this session you will be focusing on ways to 
measure the outcomes of adult education. Ask someone to give Beder’s 
definition of “outcome.” (“The changes that occur in learners as a result of 
their participation in adult literacy education.”) 

2. Review the work done so far in this inquiry process using newsprint: 

'Prtace&i: 

In the first four sessions of this process we have: 

— established and prioritized goals 

— reviewed current documentation and determined which goals are and 
are not documented 

— decided what we are accountable for, to whom, and how 

— examined the input, processes, outputs of our program and determined 
how they do or do not address our goals 
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Explain that now you are going to return again to documentation, this time 
focusing on ways to document outcomes. You will be looking at various ways 
people have tried to document the “changes that occur in learners as a 
result of their participation in adult literacy education.” 



Activity 3. Review Harold Better’s Article 

Purpose To examine some large-scale studies and compare to your 
program 

Time 20 minutes 

Materials — Newsprint and markers 

— Large 'Tttttfrux on newsprint 

Process 

1. Introduce by saying you will begin by looking at some large-scale 
outcomes studies. 

Ask each of the groups to report on the study they focused on. They should 
briefly tell: 

— what was measured in the study 

— how it was measured 

— the results. 

(FYI: Summary of studies for the facilitator) 

• The 1973 National Evaluation measured learning gains using the TABE. The 
results showed .5 grade gain in reading after 98 hours and .4 after 66 hours. In 
math there was a .3 grade gain after 98 and 66 hours. Of the 1108 people who 
were pretested, only 441 received a post-test. 

• The GAIN study measured learning gains of welfare recipients using the 
TALS.They tested a control group who were not assigned to a class as well as a 
group who had an average of 251 hours of instruction. The learning gain was not 
significant. There were substantial differences between the treatment and control 
groups from county to county, with the control group doing significantly better 
than the treatment group in two counties. 
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• The National Even Start Evaluation used a variety of data including the CASAS 
for learning gains. The part of the study using an experimental design (a control 
group) did not show a significant difference. The other component of the study 
showed a small but significant gain on the CASAS after 70 hours of instruction. 

• The National Evaluation of Adult Education Programs (NEAEP) used CASAS and 
TABE test results for 614 learners in order to assess learning gains. Statistically 
significant gains were found after the average of 15 weeks of ABE attendance. 

2. Ask someone to summarize Beder’s findings. (That outcomes studies 
show little tested learning gains, but that students report important learning 
gains.) Briefly discuss: 

— What does your program measure? (You may want to refer to the 
Documentation Matrix .) 

— How are learning gains measured? Program-wide testing using what test? 
In-class assessments? 

— What are the learning gains results for you program? (If the group doesn’t 
know, ask for a volunteer to try to find out. You may not be able to 
access this information, depending on what is reported and what data 
comes back from the state.) 



Activity 4. Outcomes as Indicators of Success 

Purpose To examine using indicators to measure goal achievement 
Time 10 minutes 

Materials — Workforce Investment Act reading from Session 1 
— Newsprint with food ‘ladicatona. list 

Process 

1 . Ask the group to look at the chart on the second page of the 

Workforce Investment Act handout. Point out that the first column 
lists the goals of WIA and the second column lists the indicators by which 
achievement of the goals will be determined, i.e., these improvements, 
placements, or receipt of diplomas will be accepted as indicators that the 
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goals in the first column are achieved. The third column lists the measures 
that are to be reported and are very similar to the indicators. The outcomes 
or changes in column two are indicators. 

2. Post the 'htdic<iton& list. Tell the group that this list was created by 
Maureen Hart, a researcher in the Quebec-Labrador Foundation/Atlantic 
Center for the Environment in Ipswich, Massachusetts. 

inclicat&ui one: 

• Relevant 

• Understandable to the community at large 

• Developed by and accepted by the community 

• Focus on a long-term view 

• Based on reliable, easily available information 

• Available on a regular basis 



Briefly discuss as a group: 

• The extent to which the WIA Core Indicators meet these criteria. 

• Which goals do not have Core Indicators? 

• Why might that be ? 

Note the difficulty of determining good indicators. 



Activity 5. Quality-of-Life Measurement 

Purpose To introduce the concept of quality-ofdife measurement 

Time 20 minutes 

Materials — Newsprint and markers 

— Quality^ of 'Life Measurements handouts 

Process 

1. Introduce the activity by saying (or paraphrasing): 

“Many people come to adult education because they want to improve their 
quality of life. Quality of life can be defined as a ‘person’s sense of well-being 
that stems from satisfaction or dissatisfaction with the areas of life that are 
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important for him/her.’ Among potential outcomes of participation in adult 
education, ‘improved quality of life’ is sometimes named, although funders 
seldom require program to measure and report it. If a program has a goal to 
make the lives of students better as a result of their participation in ABE, its 
staff might want to develop, along other outcomes indicators, some 
indicators helpful in measuring quality of life. The purpose of the following 
activity is to help participants understand the variety of domains that 
comprise a person’s quality of life. There are many ways to think about 
quality of life and to measure it.” 

2. Pass out the Quality of Life Measurement forms. Ask each participant to 
individually rate these “quality of life indicators.” In the middle column of 
the table, they should rank how IMPORTANT this indicator is for them, 
for example, “1” for the first indicator would mean, “It does not matter to 
me whether my place is as clean and comfortable as an average room in our 
local hospital,” while “4” would mean, “It is very important to me that my 
place is as clean and comfortable as an average room in our local hospital.” 
In the next column, the participant should rank the ACTUAL 
SITUATION reflecting this indictor in their life, regardless of how 
important it is to them. For example, “1” in this column for the first 
indicator would mean, “My place is a lot less clean and comfortable than 
an average room in our local hospital,” while “4” would mean, “My place is 
more clean and comfortable than as an average room in our local hospital.” 

3. After they have finished individual rating, ask everyone to share 
theirs with a partner. The pairs should compare their ratings and discuss 
why certain things were more or less important to them. 

4. After the work in pairs, start a general discussion on which 
indicators would be important for their students and would reflect 
their lives. Which indicators should you add to the list? Which indicators 
could be affected by participation in an educational program? How might 
they be measured? 
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Activity 6. Local Documentation 

Purpose To examine a documentation process developed by a local 
program 

Time 15 minutes 

Materials DoSetMet form 

Process 

1. Hand out the DoSetMet form, explaining that it is an example of local 
documentation developed by a program in Virginia as a way to document 
outcomes in their program. Instead of having indicators of outcomes, the 
form relies on student self-report (and some teacher observation) that they 
have met their goals. The achievement of the particular goal is the 
outcome. 

2. Ask the group members to complete the form for themselves, first 
checking off the items they can DO and then checking any items that they 
might SET as a goal. 

3. Ask group members to look at the goals they have SET and to 
consider what might be used as evidence that they have MET that 
goal. Ask them to try to think of a way they could document achievement 
of that goal. Go around the group and ask for examples of goals and possible 
evidence of achievement of that goal. 



Activity 7. Documentation for Your Program 

Purpose To begin to identify approaches to document your program 
outcomes 

Time 15 minutes 

Materials — Newsprint and markers 

— Large “Do&wteittattiHt ‘Tffefcix 

— 'Ptuvuty foKiti. list 

— {food ‘InduMtona. list 
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Process 

1 . In this activity the group will try to identify indicators for the goals 
they have set as priorities. First post the P'Uonity {fart*. list and the 

‘Tftatrux and ask the group to determine for which goals your 
program is collecting data that could indicate that the goal is being met. List 
the goals and the indicators that you are using on newsprint. 

2. Now list a goal for which you are not documenting any indicators. 

Ask the group to brainstorm possible indicators (using the tfayd 

list as a reference) for that goal. Do as many goals as you have time for. 

Save this list for possible use by the program. 



Activity 8. Wrap-Up 

Purpose To prepare for the next session 

Time 5 minutes 

Materials — Handout of article “A Story of Improvement” - Reading #4 
— Feedback Form 

Process 

1 . Tell the group that in your next and final session you will be 
reviewing the work you have done and considering ways your program 
might build on your work. You will be examining three approaches to 
building a quality program: Equipped for the Future which is based on 
content standards, the TESOL program standards, and the Malcolm 
Baldrige criteria for educational excellence. The article that you are giving 
them was written about a program that used Equipped for the Future and 
the Baldrige criteria. As they read the article they should consider how what 
the Knox County program did is or is not relevant to your program. 

2. Hand out Feedback Forms and complete. 
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Considering Next Steps 



The objectives of Session 6 are for the group: 

• to understand three possible approaches to on-going program 
improvement 

• to determine next steps your program might take regarding program 
improvement. 

Preparation 

The three main tasks of this session are: 

1 . to conduct a needs assessment to identify questions about your program’s 
performance; 

2. to explore several possible resources for further inquiry; and 

3. to make initial plans about next steps. 

These tasks tie in with an overall goal of this Inquiry Guide; to help you 
examine performance accountability in your program and identify directions 
for program improvement. In this guide, performance accountability is defined 
more broadly than just “official” accountability to the agencies to whom your 
program is required to report. It is accountability to your students, to your 
program as whole, and to yourselves as teachers committed to improving how 
you serve your customers. Sometimes there are no external rewards or even 
acknowledgements for these improvements; it is your own professionalism that 
is the driving force for seeking ways to advance the quality of your program. 
The activities included in this session are designed to help you to tie together 
everything that you have learned in the course of your inquiry and to provide 
direction for your next efforts. 
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Newsprint pad, markers, tape 

Agenda 

Materials used/developed in the course of the previous five sessions 

(newsprints and/or copies): Prioritized goals, Documentation Matrix, 
Accountability Grid, Inputs-to-lmpacts Model, Next Steps list 

Handouts (copies for each group member, hole-punched): 

Reading #5 - “Some Approaches to Improving Program Performance” 

Newsprint 

Questions in Activity 3 
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Activity 1 . Introduction to Final Session 

Purpose To give the group an overview of the activities for the day 
Time 5 minutes 

Materials Agenda 
Process 

1. Introduce the work of the session by saying something like: 

“Today we are going to conclude this inquiry process in which we examined 
performance accountability in our program and begin to identify directions 
for program improvement. In this guide, performance accountability is 
defined more broadly than just ‘official’ accountability to the agencies to 
whom our program is required to report. It is accountability to our students, 
to our program as whole, and to ourselves as teachers committed to 
improving how we serve your customers. Sometimes there are no external 
rewards or even acknowledgments for these improvements; it is our own 
professionalism is the driving force for seeking ways to advance the quality 
of our program. The activities included in this session are designed to help 
tie together everything that we have learned in the course of our inquiry 
and to provide direction for our next efforts.” 

2. Briefly review the day’s agenda: 

a. a “needs assessment” to identify our questions about our program 

b. a review of some possible approaches to planning and making program 
changes 

c. plans for next steps. 



Activity 2. Review of “A Story of Improvement” 

Purpose To draw on the experience of the Knox County Literacy Program 

Time 10 minutes 

Materials — “A Story of Improvement” 

— Newsprint and markers 
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— Newsprints labeled “Ktt&x P'toyiam. 

— Newsprints with questions for Activity 3 



Process 

1. Ask the group to take out their copies of “A Story of Improvement.” 

Tell them that “A Story of Improvement” describes how the Knox County, 
Tennessee, Adult Literacy Program attempted to institute a continuous 
improvement process. The section of this article called “Forces for Change” 
(p. 2), in particular, outlines the factors that prompted the program’s 
decision to start the performance assessment and continuous improvement 
processes. Ask the group to look through the article one more time and 
identify those factors. As they identify them, write them on a newsprint 
labeled "TCmxx ’P'uHyuznt. 

[FYI: the factors listed in the “Forces for Change” section are: (1) need to 
evaluate the partnership with the Friends of Literacy; (2) need to clarify the vision, 
mission, and goals; and (3) as a result of involvement with “What Works Literacy 
Partnership” and the “Equipped for the Future,” focusing on the ways to 
document and measure results of improving students’ basic skills]. 



Activity 3. A Quick Needs Assessment 



Purpose To answer questions about program performance and identify the 
next steps 

Time 30 minutes 



Materials — Newsprint and markers 

— Newsprints with questions 

— "Pnlofiitcf 

— "Tfta&Ux 

— ‘lufacti.-to-- e ]mjxxcti. "Wtadel 

— tytid 
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Process 

1. Post all the materials that you have used/developed in the course 

of the previous sessions, that have to do with program performance 
documentation and assessment ( PrUonity Z><uMm^tt<sUuut ‘7fCsttrux\ 

tyud). Ask the group to look over them 
and identify areas where improvement may be needed. 

2. After you have reviewed these materials, post newsprint with the 
following questions and answer as a group: 

fZcceatitma,: 

— Do you have consensus as a program/staff about your program goals 
(other than those that are the basis for the official reporting)? What else do 
you need to do? 

— Do existing curricular, instructional, and assessment activities/processes 
support these goals? Are there areas where additional work is needed to 
align the activities/processes with the goals? 

— Does existing documentation enable you to know whether you met your 
goals? What areas would need additional work for the documentation to be 
more helpful? 



3. Write answers on the newsprint sheet for use in Activity 5. 



Activity 4. Approaches to Improving Your 
Program Performance 



Purpose To introduce three possible approaches that your program might 
use to address the needs identified in Activity 3 

Time 30 minutes 



Materials — Newsprint and markers 

— Reading #5 - Some Approaches to Improving Program 
Performance 
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Process 

1. Pass out Reading # 5 - Some Approaches to Improving Program 
Performance. Tell group that you will briefly review each approach. Do 
that, using the first page of the handout (reproduced below) as you present. 
You may want to ask one person to read a paragraph aloud and then, as a 
group, look at the appropriate pages of the handout and discuss. 



Some Approaches to Improving Program Performance 

Improving program performance can be addressed using a variety of 
approaches — or a combination of approaches as in Knox County. In deciding what 
approach to take, programs need to consider time available, support available, 
and how an approach matches their style and values. 

The following three approaches are all being used nationally, but level of support 
for each vary from state to state. Any approach to improving program quality will 
only work if a program can commit time and focus to the process. 

Program Quality Standards. For many years the federal and state adult 
education systems have promoted the use of program standards. States and 
other groups have developed Program Quality Indicators that programs can use to 
evaluate the various components of their program and to plan change. In this 
handout we have included information on program standards for ESOL programs 
recently developed by TESOL (Teachers of English for Speakers of Other 
Languages). Their Quality Indicators describe what TESOL considers a quality 
ESOL program. The TESOL Program Standards do not describe the process for 
improving practice, but do provide a list of indicators of a quality program as 
developed by the TESOL task force. Even Start has developed a list of program 
quality attributes for family literacy program, and many states have indicators of 
program quality. To use these quality indicators a program might make an 
indicator-by-indicator assessment of “how they are doing” and then plan for the 
changes that would improve performance on particular indicators. 

Content Standards. While TESOL developed program standards, Equipped for 
the Future (EFF) is based on content standards. Content standards address what 
is taught and learned. Many states have developed content standards for subjects 
in the K-12 curriculum. The sixteen EFF content standards are part of a framework 
developed through a process involving students, practitioners, and thousands of 
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stakeholders in the adult literacy and lifelong learning system. The framework 
identifies adults’ purposes for literacy and what adults need to know to carry out 
their responsibilities as workers, family members, and community members. The 
EFF content standards focus on the application of skills in adults’ lives. This 
handout includes a brief introduction to EFF, the “standards wheel” that identifies 
the sixteen standards, and an example of one standard. The EFF initiative of the 
National Institute for Literacy is developing as assessment framework to 
accompany the standards. The EFF framework and standards do not provide a 
specific set of indicators in the same way that program standards do. But the 
framework does provide a variety of tools including a guide to EFF approach to 
quality that programs can use to examine their practices and to plan and 
implement changes. 

Continuous Improvement. The Malcolm Baldrige Educational Criteria for 
Performance Excellence are based on total quality management principles 
originally developed in the business sector. The Baldrige criteria outline an orderly 
process for examining and improving program operations. They provide the 
questions to ask but not the answers. Where program and content standards 
define quality, the Baldrige criteria focus on processes to build quality. This 
handout includes a graphic presentation of the process and a review of the key 
items in each category. 



2. Ask the group to take about 10 minutes to look through the 
handout (if you did not do this as you discussed). 



Activity 5. Next Steps for Your Program 

Purpose To make plans on how (and if) to follow up on the inquiry 
process in your program 

Time 15 minutes 

Materials Newsprint and markers 
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Process 

1 . Tell the group that you are going to answer a set of questions to 
determine next steps. The first question is whether or not you have 
identified areas where you need to improve as a program. Look back at the 
three newsprints you generated in Activity 3. Ask the group if there are 
there areas your program could improve? If so, do you see things that your 
program could take action on? 

If the answer to either of these are “No,” then you are done. Thank people 
for all their participation. 

If they answer “Yes,” go on to step 2. 

2. You now need to determine, in a group discussion what should be 
your next steps. Begin by making a succinct list of items that need 
attention, referring to the newsprints from Activity 3 and to your earlier 
work. This should be a list that you share with the rest of your program. 

3. Determine who from your program should be involved in carrying 
out these next steps and who from your inquiry group will talk with 
them. 

4. Determine if you want to investigate any of the approaches in 
Activity 4 and who will do it. 

5. Decide who will meet again and when. 

6. Thank the group and celebrate! 
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Participants’ Materials List 

— Introductory letter to Adult Educators concerned about program improvement 

— Overview of the Inquiry Process 

— Schedule/To Do Form 

— Feedback Form 

Session One 

— Agenda 

— Goals Sheet 

— Reading # 1 - The Workforce Investment Act: A Brief Introduction 

Session Two 

— Agenda 

— Documentation Matrix 

— Reading # 2 - Performance Accountability: For What? To Whom? And How? 

Session Three 

— Agenda 

— Accountability Grid 

Session Four 

— Agenda 

— Inputs-to-lmpacts Model 

— Sample Inputs-to-lmpacts Model 

— Glossary 

— Reading # 3 - Lessons from NCSALL’s Outcomes and Impacts Study 

— Worksheet for Reading # 3 

Session Five 

— Agenda 

— Quality of Life Measurement 

— DoSetMet Form 

— Reading # 4 - A Story of Improvement 

Session Six 

— Agenda 

— Reading #5 - Some Approaches to Improving Program Performance 
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To Adult Educators Concerned About Program Improvement: 

What difference are <we making ? How do we know ? How can we show it? 

These are questions asked by adult literacy and ESOL teachers, program administrators, 
and funders. You are invited to take part in a process designed to facilitate a systematic 
inquiry into these kinds of questions. In an inquiry process, organized into six sessions, you 
will identify and clarify program goals, examine current documentation processes, and 
begin to plan changes to better address the challenges of performance accountability and 
outcomes documentation in your program. 

An inquiry process is a systematic investigation of a particular issue. This inquiry process 
will produce a number of documents that a program can use a) to make a decision about 
implementing on-going improvement work and b) in conducting this ongoing work. 

The staff that participates in this inquiry process will have a clearer understanding of: 

I. Their goals, their program’s goals, and state and national goals 

II. How to document the accomplishment of goals 

III. Performance accountability and what it means 

IV. How the various parts of their program fit together to move toward goals 

V. Possible next steps their program can take to move more effectively toward their goals. 

The activities in the sessions include: 

• looking at program work — goals, program activities, and documentation processes 

• reading and discussing articles on performance accountability 

• examining state and national plans for the fits (and misfits) with your program 

• beginning to develop possible next steps for program change. 

If you decide to participate as a member of this inquiry group, you will be asked to 
participate in six sessions, each lasting about one and half hours. You will also need to do 
some work in preparation for Sessions 2-6; the preparatory activities and reading should 
take about an hour a week. For this investment of your time and thinking, you will gain a 
clearer understanding of how your program works and why. You will also have the 
opportunity to contribute to process of program improvement that will enable you to better 
meet your goals and the goals of your students and program. 

For more information about how to get involved in this exciting inquiry process, contact: 
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An Overview 



What difference do <we want to make ? What difference are we making ? 

How do we know? How can we show it? 

This inquiry process is designed to help groups of adult education staff begin to 
answer these questions. The process is divided into six sessions described below. 

Session 1: Examining Our Goals 

The objectives of Session 1 are for the group: 

• to get to know each other and the process 

• to examine and clarify student, staff, state and federal goals. 

The activities for Session 1 are designed to introduce the process, get the group 
organized, to identify and clarify the goals of the staff, students, and program, 
and to compare local goals with state and national goals for adult education. 



Session 2: The Documentation Matrix 

The objective of Session 2 is for the group: 

• to develop a clearer understanding of their program’s current 
documentation — what is documented and why? 

The activities include looking at the various documentation forms used by the 
program, using the Documentation Matrix to analyze how they are used, and 
determining if there are program goals that are not documented. 



Session 3: Performance Accountability 

The objectives of Session 3 are for the group: 

• to have a greater understanding of performance accountability in adult 
education 

• to consider how and to whom their program is accountable. 

The activities for Session 3 include completing the Accountability Grid to 
analyze performance accountability in your program, discussion of Juliet 
Merrifield’s article on performance accountability, and thinking of next steps 
toward performance accountability for your program. 
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Session 4: Inputs-to-lmpacts 

The objectives for Session 4 are for the group to: 

• to prioritize their program goals 

• to analyze how the components of their program address program goals. 

The activities for Session 4 include prioritizing the goals identified in Session 
1, reviewing the components of your program using the Inputs-to-lmpacts Model, 
and identifying any gaps in addressing your goals. 



Session 5: Documenting Outcomes 

The objective for Session 5 are for the group to: 

• to better understand outcomes documentation 

• to determine the outcomes that might be documented to assess program 
goals. 

The activities for Session 5 include discussing Hal Beder’s review of outcomes 
studies, identifying the outcomes that need to be documented in order to assess 
performance in meeting program goals, and examining several approaches to 
outcomes documentation. 



Session 6: Planning Next Steps 

The objectives for Session 6 are for the group to: 

• to understand three possible approaches to on-going program improvement 

• to determine next steps your program might take regarding program 
improvement. 

The activities for Session 6 include reporting on and discussing readings on 
approaches to program improvement, identifying gaps in current program 
processes, and identifying possible next steps the program might take toward 
continuous improvement. 
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Schedule/To Do Form 

What to do to get ready 



Session 


Date 


What to do before session 


Session One 




[If you receive before Session One — review 
readings and highlight interesting points and 
questions.] 


Session Two 




Collect and bring documentation you use in 
class and program — tests, forms, etc. 


Session Three 




Read Merrifield article (Reading # 2) and write 
one question the article has raised for you. 


Session Four 




Review Goals list with students and 
other staff. 


Session Five 




Read Beder article (Reading # 3) and 
complete worksheet. 


Session Six 




Read “A Story of Improvement” (Reading # 4). 
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Feedback Form 



Please complete and return to the facilitator. 

What questions do you have from this session? 



I want the facilitator to know... 



I think I can use what I learned today to... 
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Session 1: Agenda 

1 . Introduction — to people and the process 

2. Getting organized — schedule, ground rules, keeping it together 

3. Review of our goals 

4. Looking at state and national goals 

5. Feedback and getting ready for the next session 

Reflections on today’s session 
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Goals Sheet - Identifying Our Goals 



Staff 


Students 


Program 
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The Workforce Investment Act: A Brief Introduction 

— by Mary Beth Bingman 



The purpose of the Adult Education and Family 
Literacy Act, Title II of the Workforce 
Investment Act, is to provide adult education 
services in order to: 

( 1 ) assist adults to become literate and obtain the 
knowledge and skills necessary for 
employment and self-sufficiency; 

(2) assist adults who are parents to obtain the 
educational skills necessary to become full 
partners in the educational development of 
their children; and 

(3) assist adults in the completion of a secondary 
school education. (P.L. 105-220) 

Language about meeting personal goals or 
developing knowledge, which was included in the 
1991 legislation, is absent in WIA. 

WIA mandates a performance accountability 
system in order “to assess the effectiveness of 
eligible agencies in achieving continuous 
improvement of adult education and literacy 
activities funded under this subtitle, in order to 
optimize the return on investment of Federal 
funds in adult education and literacy activities’ 
(P.L. 105-220). States are required to set levels of 
performance for three core indicators: 

(i) Demonstrated improvements in literacy skill 
levels in reading, writing, and speaking the 
English language, numeracy, problem solving, 
English language acquisition, and other 
literacy skills. 



(ii) Placement in, retention in, or completion of 
postsecondary education, unsubsidized 
employment or career advancement. 

(iii) Receipt of a secondary school diploma or its 
recognized equivalent. (P.L. 105-220) 

The National Reporting System 

The National Reporting System for Adult 
Education has been developed by the American 
Institutes for Research under contract to the 
Division of Adult Education and Literacy. This 
system establishes measures for the core indicators 
required by the Workforce Investment Act. The 
development of the NRS began as the National 
Outcomes Reporting System Project. That 
project (begun before WIA was enacted) was 
developed in response to the concerns of state 
directors of adult education about performance 
accountability. The original system was to focus 
on seven categories of outcomes endorsed by state 
directors: economic impact, credentials, learning 
gains, family impact, further education and 
training, community impact, and customer 
satisfaction (Condelli & Kutner, 1997). 

The National Reporting System now being 
implemented focuses on the three core indicators. 
The measures to be reported include: educational 
gains, entering or retaining employment, and 
placement in postsecondary education or passing 
the GED test (NRS, 1999). In addition to the 



Excerpted from Outcomes of Participation in Adult Basic Education: The 
Importance of Learners' Perspective , NCSALL Occasional Paper, by Mary Beth 
Bingman with Olga Ebert & Brenda Beil. The full paper can be downloaded from 
http://gseweb.harvard.edu/~ncsall/research/occas.htm 
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outcome measures, the NRS will collect data on 
descriptive measures (student demographics, 
reasons for enrolling, and student status) and 
participation (contact hours and enrollment in 
special programs). 

As can be seen in the NRS graphic reproduced 
below, the attainment of the goals of the Adult 
Education and Family Literacy Act is to be 
measured in large part by the achievement of basic 
academic skills that are measured in most states by 
standardized tests. The limited connection 
between goals and indicators has happened in part 
because Congress did not establish core indicators 
for family literacy. And while the purposes of 
WIA refer to the “knowledge and skills necessary” 



for employment and self-sufficiency, these were 
not defined. 

Educational gains are to be measured by 
movement from one to another of six 
“educational functioning levels” which include: 
Beginning ABE Literacy, Beginning Basic 
Education, Low Intermediate Basic Education, 
High Intermediate Basic Education, Low Adult 
Secondary Education, High Adult Secondary 
Education. The ESL levels are: Beginning ESL 
Literacy, Beginning ESL, Low Intermediate ESL, 
High Intermediate ESL, Low Advanced ESL, and 
High Advanced ESL (NRS, p.6). For each level 
the NRS has identified descriptors in three areas: 
Basic Reading and Writing, Numeracy Skills, and 



Goals and Core Indicators of WIA 

Adult Education and Family Literacy Act and NRS Core Outcome Measures 



Goals of Adult Education 
Described in the Adult 
Education and Family Literacy 
Act of WIA 


Core Indicators Required by 
the Adult Education and Family 
Literacy Act of WIA 


National Reporting System 

Core Outcome Measures 


Assist adults to become literate 
and obtain the knowledge and 
skills necessary for employment 
and self-sufficiency. 

Assist parents to obtain the 
skills necessary to be full 
partners in their children’s 
educational development. 


Improvements in literacy skill 
levels in reading, writing and 
speaking the English language, 
numeracy, problem-solving, 
English language acquisition, 
other literacy skills. 


Educational gains (achieve 
skills to advance educational 
functioning level) 


Placement in, retention in, or 
completion of, postsecondary 
education, training, 
unsubsidized employment or 
career advancement. 


Entered employment 
Retained employment 
Placement in postsecondary 
education or training 


Assist adults in the completion 
of secondary school education. 


Receipt of a secondary school 
diploma or its recognized 
equivalent. 


Receipt of a secondary school 
diploma or pass GED test 



(DAEL, 1999, p. 6) 
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Functional and Workplace Skills for ABE; and 
Speaking and Listening, Basic Reading and 
Writing, and Functional Workplace Skills for ESL. 
For example, the ability to complete medical 
forms is a functional skill at the Low Intermediate 
Basic Education Level, and reading text about and 
explaining the use of a complex piece of 
machinery is a workplace skill at the High Adult 
Secondary Level. The ability to participate in a 
conversation in some social situations is a 
descriptor for speaking and listening in the High 
Intermediate ESL level. 

Placement of adult learners in these levels is 
conducted at enrollment and again after pre- 
determined time periods while the adult is an 
active student. All programs in a state must use 
the same assessment procedure. While the NRS 
states “T heir inclusion in no way is meant to imply 
that the tests are equivalent or that they should be used 
as the basis for assessment” [italics in original] 

(NRS, p. 19), test benchmarks are provided for 
each educational functioning level. The NRS 
assumes that increases in scores on standardized 
tests like the Test of Adult Basic Education 



(TABE) or Comprehensive Adult Student 
Assessment System (CASAS) will indicate 
increases in skills and, therefore, in functioning 
levels. 

Employment and postsecondary education and 
training measures and receipt of a secondary 
credential are follow-up measures to be collected 
after students leave a program. These measures 
are collected only for those adults whose goals 
include employment, further education, or a 
secondary credential. States may use either survey 
or data matching (using Social Security numbers) 
methodologies for the follow-up measures for 
employment and secondary credentials. 



References 

Division of Adult Education and Literacy, U.S. Department 
of Education (1999). Measures and methods for the 
National reporting System for Adult Education. 

Washington, DC: Author 

Workforce Investment Act - Title II. (1998). Public Law 105- 
220- Aug. 7, 1998. 112 Stat. 936. Washington, DC: U.S. 
Congress 
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Session 2: Agenda 

1 . Check In 

2. Sorting documents 

3. Completing the Documentation Matrix 

4. Determining which goals are documented 

5. Feedback and getting ready for the next session 

Reflections on today’s session... 
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Documentation 

Method 












Who Does It? 












For Whom? 












How Often? 












How Used? 












Key Items Reported 
(Themes) 



Documentation Matrix 

(Developed by the NCSALL Documenting Outcomes project) 
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Reading #2 

Performance Accountability: For What? To Whom? And How? 

— by Juliet Merrifield 



In everyday life, accountability means 
responsibility; it means being answerable to 
someone else for one’s actions. We cannot, 
however, use the term without specifying 
accountability to whom and for what. In adult 
basic education (ABE), how we answer the 
question “to whom” depends a lot on our position 
in the system. Teachers may answer that they feel 
answer accountable to their students. Program 
directors may that they are accountable to their 
funders and staff as well as to students. State adult 
education offices may feel accountable to the 
governor, the legislature, to other state agencies, 
to workforce development boards, as well as to 
taxpayers. In addition, no clear consensus exists 
about “for what” adult education is accountable. 
Where does the balance lie between providing 
services and delivering results? Is the main purpose 
increased literacy proficiency, or are more diffuse 
social outcomes the emphasis? Until recently, the 
focus has been on providing services, with little 
emphasis on the results or the impact of those 
services. In the last few years, a number of policy 
initiatives at state and federal levels have begun to 
shift the emphasis to delivering results, with 
services seen as the means to an end. But what 
the “end” should be is by no means clear. 

I would like to suggest that developing 
performance accountability is not just technically 
challenging but also challenges our values. The 
key issues do not have purely technical solutions. 
They require agreement on what is important to 



us, on what we want out of adult education. If 
they are to be resolved, they require involvement 
by the ABE field as a whole. 

Adult basic education is facing serious demands 
from policy-makers and funders to be accountable 
for its performance. The 1998 Workforce 
Investment Act (WIA) requires that each state 
report on performance measures. The emphasis on 
results shifts attention from simple delivery of 
services to the outcomes of learning: learning 
gains measured on standardized tests or social and 
economic outcomes such as getting a job, getting 
off welfare, and children’s school success. 

The key issues in the development of performance 
accountability in adult education are: 

• What does good performance mean? 

• Do programs have the capacity to be 
accountable? 

• Are the tools commonly used for measuring and 
documenting performance adequate and useful? 

• Are accountability relationships in place to link 
ABE into a coherent system? 

Good Performance 

Accountability systems work best if stakeholders 
— those who have an interest in the outcomes of 
the system — agree on what success looks like. For 
adult basic educators, the heart of the matter is 



Article reprinted from Focus on Basics, Voi. 3, Issue B, June 1999 and based 
on Contested Ground: Performance Accountability in Adult Basic Education, 
NCSALL Report #1 by Juliet Merrifield. The full report can be downloaded from 
http://gseweb.harvard.edu/~ncsall/research/reports.htm 
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our concept of literacy. That concept has shifted 
over time from reading and writing text to 
functioning in society, from a simple dichotomy of 
illiterate/literate to multiliteracies. Brian Street 
characterizes two broad conceptual notions of 
literacy. The autonomous model conceives of 
literacy as a discrete and fixed set of skills, 
transferable from one context to another. The 
ideological model conceives of literacy as practices 
that are sensitive to social context and inherently 
associated with issues of power and access (Street, 
1984). 

Much recent research on multi-literacies suggests 
that there are multiple purposes for literacy and 
multiple goals and expectations for literacy 
education (Heath, 1983; Barton, 1994; Street, 
1984,1995; Lankshear, 1997; New London Group, 
1996). In such an understanding, notions of 
success must also be multiple. A single definition 
of success — gaining the GED, for example, or 
getting a job — excludes learners who have 
different purposes. 

Definitions of success should be negotiated among 
all the stakeholders, learners, and practitioners as 
well as policymakers and funders. Although the 
legislative goals of the Workforce Investment Act 
reflect a majority among lawmakers, other 
stakeholders — including policy makers, program 
managers, teachers and students — may focus on 
other purposes for adult education and look to 
other measures of good performance. 

Next Steps: Agree on Performance 

Practitioners can play a role in defining 
performance within their own states. The WIA 
requires that each state develop a plan of the 
performance measures it will use to track results, 
including but not limited to those required by the 



Act. Whether explicitly or implicitly, these 
measures will define what counts for the field. The 
challenge is to come to an agreement on 
performance that includes the full diversity of 
learner and societal purposes. Lessons from the 
literature and experience in education and other 
fields suggest states should: 

• Invest time and energy in agreeing on what 
performance means; 

• Involve stakeholders and seek consensus; 

• Reflect newer understandings of literacy and 
connect performance with real life; and 

• Acknowledge a variety of outcomes as 
acceptable performance, as a way of including 
the full diversity of learners and programs. 

Capacity to be Accountable 

Adult education is trying to develop a national 
accountability system without having developed 
the capacity of the service delivery system to 
document and report results (Moore & Stavrianos, 
1995). Plenty of evidence documents the lack of 
valid, reliable, and useful data about performance 
(Young et al., 1995; GAO, 1995; Condelli, 1994). 
These studies suggest some of the most basic data 
are absent, incomplete, or of low quality. 

When asked to report numbers, programs will 
indeed report numbers. But as the GAO report on 
adult education says, “the data the Department 
receives are of questionable value” (GAO, 1995, 
p. 33). This is not surprising, since staff in 
programs usually do not use the data, rarely see 
reports based on them, and see no one else placing 
any real value on them. 

Performance accountability requires investment in 
the ability of local programs to collect, interpret, 
and use data to monitor how well they are doing. 
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A number of states such as Pennsylvania, 
Connecticut, and Arkansas have already begun to 
develop their capacity for accountability. (For an 
overview of Pennsylvania’s program, Keenan’s 
article.) They consistently learned from their 
experiences that the key is to get buy-in from 
programs and practitioners from the beginning 
(Merrifield, 1998). They are also acutely aware of 
the problems of deciding what is counted, as well 
as how it is counted. 

What is counted becomes what counts. Many 
examples of the hazards of counting the wrong 
things exist. A healthcare delivery system 
emphasizes cutting the numbers of people on a 
waiting list for surgery, thus ensuring that people 
with minor needs get served quickest because 
more operations for varicose veins than for heart 
bypasses can be performed in one day. The 
original performance standards of the Job Training 
Partnership Act (JTPA), an education and 
training program, emphasized the numbers of 
people placed in jobs within a specific time frame. 
This ensured that programs recruit clients who 
were most qualified and therefore easiest to move 
into jobs quickly and cheaply (GAO, 1989). 

Next Steps: Build Capacity 

Two kinds of capacity — to perform and to be 
accountable — are linked (Merrifield, 1998). By 
instituting a learning organization approach with 
feedback loops, performance data can help 
programs improve performance and increase 
accountability. Building the capacity to perform 
involves: 

• Increasing resources and focusing them on 
quality rather than quantity; 

• Providing staff development and training and 
technical support; 



• Using performance data for continuous 
improvement. Building the capacity to be 
accountable involves ensuring that: 

• Accountability demands are commensurate with 
resources and capacity; 

• Users of measurement tools are engaged in their 
development; 

• Staff training and support are provided; 

• Information is timely; 

• Improved performance is rewarded. 

A variety of efforts are already underway to build 
capacity to perform and to be accountable. 
Teacher inquiry projects have involved individual 
teachers in examining their practice and 
identifying ways to change and develop (Smith & 
Lytle, 1993). Some programs, such as those 
described elsewhere in this issue, have been 
working on their capacity to use data for 
continuous improvement. Some states have begun 
efforts to build local program capacity for both 
performance and accountability. The National 
Accounting and Reporting System (Condelli, 
1998), will be providing training and support on 
how to use newly revised WIA-related 
performance measures. 

Accountability Tools 

For accountability purposes, it is crucial that we 
collect data that are relevant, adequate, and 
important. To do so, we need tools — indicators 
and measures — that we believe in and use well. 
Indicators and measures are approximations of 
reality, not reality itself. They can be good, bad, 
and indifferent. An indicator that measures 
something unrelated to literacy learning — the 
number of brown-eyed learners, for example — is 
irrelevant. An indicator that measures something 
relevant — the prior learning that students bring, 
for example — but in an inadequate way, is 
dangerous. An inability to measure something 
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important — affective changes in learners, for 
example — can be disastrous. 

Some of our current accountability tools are 
inadequate: what we use to measure literacy gains 
is one example. Standardized tests are widely used. 
While such tests have their uses for placement 
purposes, their validity as measures of performance 
is questionable (Venezky, 1992). “The research 
literature raises questions about the validity of 
standardized tests... and local program staff have 
questioned the appropriateness of using these 
assessment to measure program results” (GAO, 
1995, p. 24). As yet, however, few alternatives to 
standardized tests exist. Some programs are using 
various tools, such as portfolios, that allow 
learners to demonstrate their learning 
authentically (Literacy South, 1997), but so far 
these cannot compare learning between learners 
and across programs. Without external criteria or 
standards, authentic assessment will not meet the 
needs of accountability systems. 

How we collect data for accountability is also 
important. Different approaches to data collection 
and analysis meet different purposes. A complete 
performance accountability system would include 
several approaches: monitoring, evaluation, and 
research would all have a place. 

Monitoring can answer ongoing questions about 
day- today program operations. What kinds of 
students are being recruited? How long are they 
staying? What do they say they want from their 
learning experiences? How satisfied are they with 
the program? Monitoring is part of everyday 
management, providing a routine way for program 
staff to see how well the program is working. 
Evaluation can answer particular questions about 
program operations at particular points of time. 
How are learners being served? Are they making 



progress on their learning goals? Is the program 
meeting quality standards? Evaluation may include 
a look at program-monitoring data. It may also 
involve gathering new data to answer specific 
questions. Surveys or focus groups are useful 
evaluation techniques. 

Research can answer questions about associations, 
correlations, and meaning, and often takes a 
broader focus than one program. Research 
questions might examine: What are the benefits 
to individuals and society of participation in adult 
education? Which program designs are associated 
with different results? What kinds of resources are 
needed to support specific program designs? 
Research may be conducted by outside researchers 
or by practitioners themselves (Quigley & Kuhne, 
1997). 

Each of these accountability technologies 
illuminates different aspects of reality. They have 
different strengths and need to be used 
appropriately. Carrying them out involves scarce 
resources, so they should be applied carefully and 
economically to ensure that the data collected are 
both useful and used. 

Next Steps: Develop New 
Measurement Tools 

New approaches and tools for measurement are 
needed that are linked to performance. 
Performance assessment tools enable us to assess 
literacy practices. For accountability purposes, this 
more authentic assessment of literacy practices 
demands that we develop external standards or 
criteria against which individual student learning 
can be measured, and through which program 
performance can be assessed. Initiatives in 
performance assessment in countries such as 
Britain and Australia may provide useful models 
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for measuring and assessing learning. We should 
use the full potential of research, evaluation, and 
monitoring technologies to meet the needs of 
different stakeholders. 

Mutual Accountability 

Underlying all the other issues in performance 
accountability for ABE is the question of 
accountability relationships. Traditional 
approaches to accountability echo Taylorist 
manufacturing systems, in which quality control 
checks at the end of the production line ensure 
that widgets meet product specifications and 
accountability runs only one way. Assessing 
outcomes at the end of the production process has 
its place in quality control systems, but 
increasingly businesses are turning to more 
participatory approaches to managing work 
processes and using production data for 
continuous improvement (Stagg, 1992). 

High performance workplaces build in processes at 
each stage of production to monitor and improve 
performance. They involve workers in this 
monitoring. The business world is now utilizing 
concepts such as the learning organization: one 
that facilitates the learning of its members to 
transform itself continuously (Pedler et al, 1991). 
This approach is seen as a way of responding to 
changing environments and multiple demands. 
This kind of learning and transformation has to be 
shared and internalized: it cannot be imposed 
from the top (Stein, 1993). Accountability is 
shared or mutual. 

In ABE, mutual accountability would engage 
members of the organization in creating a 
common vision, determining goals and customer 
expectations, and designing effective means of 
monitoring processes and results. Every member 



would be both accountable to others and held 
accountable by them. Learners would hold 
teachers, for example, accountable for providing 
learning opportunities that meet their needs. 
Teachers, in turn, would hold program directors 
and funders accountable for providing the 
resources they need to meet learner needs. These 
might include materials, space, training, pay for 
lesson planning and assessment. 

Spelling out relationships of mutual accountability 
reveals some that are overlooked in conventional 
accountability systems. Congress, for example, 
holds adult education programs accountable for 
providing effective and efficient services. But 
Congress should also be held accountable by 
programs, by learners, and by voters for identifying 
a social need, passing appropriate guiding 
legislation, and providing the resources needed to 
create a strong adult education system. 

Learners should hold their teachers accountable. 
But programs should also hold learners 
accountable for taking learning seriously and for 
making an effort to participate fully. 

Businesses who expect adult education to provide 
them with workers equipped with basic skills 
might be expected in turn to provide jobs for 
those workers, or to continue a workplace basic 
skills program when the grant runs out. Mutual 
accountability would require all the partners to 
honor their contracts. 

An accountability system based in the concept of 
mutuality has several characteristics: 

• It is negotiated between the stakeholders in a 
process that engages all the players in clarifying 
expectations, designing indicators of success, 
negotiating information flows, and building 
capacity. 
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• Each responsibility is matched with an equal, 
enabling right: the right to a program that meets 
one’s learning needs with the responsibility to 
take learning seriously, for example. 

• Every player knows clearly and agrees to what is 
expected of them. 

• Every player has the capacity to be held and to 
hold others accountable. 

• Efficient and effective information flows enable 
all players to hold others accountable. 

Inequalities of power and uneven access to 
information prevent the development of mutual 
accountability. Learners, for example, cannot 
become real stakeholders in mutual accountability 
until they have other ways to effect change 
beyond dropping out. They will only become part 
of the structure of accountability when they have 
real power to make choices. Some community' 
based programs encourage learner participation in 
management, with learner representatives sitting 
on boards, and being involved in management 
decisions about the program. Many statedevel 
adult learner organizations are working to address 
the inequalities in power and in access to 
information, and to strengthen the voice of adult 
learners in the system. 

How information flows is also a central issue in 
mutual accountability. Without adequate access to 
information, stakeholders cannot hold others 
accountable. In traditional information flow 
designs, information is collected at the base and 
increasingly summarized for the purposes of 
different levels on the way up: from program to 
community, state, and national levels. In this 
simplistic model, information flows only one way: 
up the system to the state and national levels. Few 
people have either access to or the ability to use 
the data. 



This model will not fit the needs of an 
accountability system that takes into account 
different performances and purposes and has 
mutuality as an underlying assumption. A more 
complex information model should allow 
information to be generated at all levels and to 
flow around the system, up, down and across it, 
among and between different players who use it 
for specific purposes at specific times. 

Next Steps: Develop Mutual 
Accountability 

Reforming accountability requires moving from 
one- way, top- down lines of accountability to a 
mutual web of accountability relationships. To 
make this switch, we must: 

• Bring the full range of stakeholder groups into 
the process — including teachers and learners; 

• Provide support for stakeholders who have least 
access to information and power; 

• Increase information flows among and between 
all stakeholders and make the information 
transparent (accessible to all); 

• Develop learning organizations at the program 
and state levels that would emphasize learning 
and continuous improvement, shared 
responsibility, and engagement in monitoring 
results. 

What Next? 

To implement performance accountability well 
requires agreement on good performance, capacity 
both to perform and be accountable, new tools to 
measure performance, and a strong system of 
mutual accountability relationships. In the 
business world, high performance is associated 
with extensive changes in organizational practices, 
including a broadly understood vision and 
mission, flatter hierarchies with decision-making 
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pushed as close to the shop floor as feasible, and 
participation at all levels of the organization in 
monitoring and improving performance. If ABE is 
to meet society’s need for high performance, it too 
needs to change. But these changes cannot be 
implemented from the top alone. They will 
require federal and state government departments 
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Session 3: Agenda 



1 . Complete Accountability Grid to clarify your program’s accountability 
system. 

2. Discuss the Merrifield article. 

3. Use Merrifield ’s ideas to consider performance accountability in your 
program. 

4. Feedback and getting ready for the next session. 



Reflections on today’s session... 
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Accountability Grid 



Accountable for? 


To Whom? 


How? 
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Session 4: Agenda 



1. Prioritizing program goals. 

2. Examining the Inputs-to-lmpacts Model. 

3. Completing the Inputs-to-lmpacts Model. 

4. Analyzing the Model. 

5. Wrap-up. 

Reflections on today’s session... 
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Inputs-tolmpacts Model 





Student 


Program 


Inputs - factors 
available for 
performance 






Processes - the 

educational and 
organizational 
processes 
contributing to 
performance 






Outputs - the 

immediate results 
of services provided 






Outcomes - the 

longer-term results 
of education for 
individuals and 
programs 






Impacts - changes 
in community 
brought about by 
changes in learner’s 
lives 
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Inputs-tolmpacts Model 

(Sample - Documenting Outcomes for Learners and Their Communities) 





Student 


Program 


Inputs - factors 
available for 
performance 


• previous • needs 

educational • temperament 

experiences (e.g., shyness) 

• life experiences • challenges, e.g., 

• goals L.D., childcare, 

• abilities transportation 

• commitment to 
learning 


• building, equipment 

• staff 

• curriculum 

• materials 

• goals 

• volunteers 

• technology 


Processes - the 

educational and 
organizational 
processes 
contributing to 
performance 


• intake interview • testing & 

• orientation assessments 

• reading, writing, • discussion/ 

math activities analysis 

• social • cultural 

interactions expressions 

• school • computer use 

governance 

activities 


• planning 

• record-keeping 

• assessment 

• scheduling 

• instruction 

• staff development 

• advice, guidance, support 

• connecting to human services, etc. 


Outputs - the 

immediate results 
of services provided 


• test scores 

• journals 

• more comfort in class 

• GED 

• resume 

• certificates 

• documentation of improved 
performances 


• number of classes offered 

• number of hours of instruction 

• number of students 

• staff development activities 


Outcomes - the 

longer-term results 
of education for 
individuals and 
programs 


• new reading, • changed/new 

writing, math skills 

practices • driver’s license/ 

• changed self- CDL license 

concept • workforce skills. 

• opened a • citizenship 

checking / • a job; job 

savings account promotion 

• computer skills • checking/ 

• new goals savings acct. 


• aggregation of student outcomes 

— test scores 

— GED 

— student goals met 

• teacher changes 

• records kept 

• improvement in program quality(PQI) 

• changes in administration philosophy 


Impacts - changes 
in community 
brought about by 
changes in learner’s 
lives 


• children more involved in school 

• increased use of public resources 

• more activity in civic life 

• pressure for improved neighborhood _____ , 

7 , .... , r This DRAFT was produced by the 

. better educated/developed workforce staff and action rese arch teams of 

the CLS NCSALL Documenting 
Outcomes Project, 1999 
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Glossary of Terms Used in the 
Inputs-tolmpacts Model 



In the model: 

Input is defined as what the student or program brings to the processes of the 
program. Examples include students background, teacher qualifications, 
facilities. 

Processes are the educational and organizational activities that take place in 
the program. Examples include instruction, assessment. 

Outputs are defined as the immediate results of these processes. 

Examples include test scores, student work. 

Outcomes are the changes that occur in students’ lives as a result of their 
participation in the program, or the long-term results of program improvement. 
Examples include use of new skills, self-confidence, civic participation. 

Impacts are the changes in the community resulting from changes in 
students and programs. Examples include increased membership in community 
organization, improved children’s academic performance in schools. 
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Reading #3 

Lessons from NCSALL’s Outcomes and Impacts Study 

— by Harold Beder 



Researchers occasionally encounter contradictory 
findings, findings that disagree with each other to the 
extent that it is hard to imagine both could be true. 
Although this is frustrating, it is exciting, too, because 
in resolving contradictions new insights often emerge. 
In NCSALL’s Outcome and Impacts Study, we were 
faced with such a contradiction. After examining 
testing data from seven outcome and impact studies, 
we concluded that the evidence was insufficient to 
determine whether adult basic education participants 
gain in basic skills. In contradiction, however, learners 
in ten studies were asked if they gained in reading, 
writing, and mathematics, and they overwhelmingly 
reported large gains. What led to this contradiction, 
and what is the answer to the gain question ? In this 
article we will examine possible reasons for the 
contradiction, but before we do, it is important to 
know something about the study. And as for the 
question of gain: that remains to be answered. 

Critical Issue 

We studied the outcomes and impacts of adult 
literacy education because the subject is critical 
for adult literacy educators today. Policy makers 
who control resources have increasingly 
demanded that accountability be based on 
program performance as measured by impact on 
learners. Indeed, under the newly enacted 



Workforce Investment Act (HR 1385), programs 
which fail to achieve stipulated outcomes can be 
severely sanctioned. And, since outcome and 
impact studies can identify program strengths and 
weaknesses, their results provide vital information 
for program planning and policy formation at the 
national, state, and local levels. 

We characterized outcomes as the changes that 
occur in learners as a result of their participation 
in adult literacy education. We saw impacts as the 
changes that occur in the family and society at 
large. Commonly studied outcome variables 
include individual gains in employment, job 
quality, and income; reduction of welfare; learning 
gains in reading, writing, and mathematics; GED 
acquisition; and changes in self-confidence. 
Common impact variables include effects on 
children’s reading readiness, participation in 
children’s school activities, and whether learners 
vote. 

The NCSALL Outcomes and Impacts Study 
examined outcome and impact studies conducted 
since the late 1960’s, including national studies, 
state-level studies, and studies of welfare, 
workplace, and family literacy programs. Its goals 
were to determine if the publicly-funded adult 
literacy education program in the United States 



Article reprinted from Focus on Basics, Vol. 2, Issue D, December 1998 and 
based on The Outcomes and Impacts of Adult Literacy Education in the 
United States, NCSALL Report #6 by Hal Beder. The full report can be 
downloaded from http://gseweb.harvard.edu/~ncsall/research/reports.htm 
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was effective; to identify common conceptual, 
design, and methodological problems inherent in 
the studies; to raise issues for policy and to make 
recommendations for research, policy, and 
practice. In essence, the research was a study of 
studies. To determine whether the adult literacy 
program was effective, we prepared case studies on 
the 23 outcome studies that we judged to be the 
most credible from a research perspective (see box 
for the criteria we used). Then, based on the case 
studies, we conducted a qualitative meta-analysis 
in which we treated each study’s findings as 
evidence that we weighed to make conclusions 
about program effectiveness on commonly studied 
outcome and impact variables. Consensus among 
these studies has pointed towards positive impact, 
and in making conclusions, we gave the evidence 
from more credible studies more weight. 

There is no consensus on what adult literacy is, 
but acquisition of skills in reading, writing, and 
mathematics is included in almost everyone’s 
definition of what adult literacy education should 
achieve. Outcome and impact studies have 
measured reading, writing, and mathematics gains 
in two ways: via tests or by questionnaires or 
interviews. Yet as we noted at the outset, the 
findings using these two methods conflict. While 
the results from tests are inconclusive on gain, 
when asked, learners generally report large gains. 
This calls into question whether tests are correct, 
whether learners are correct, or whether another 
explanation exists. To understand the lessons that 
can be learned from examining this issue, we will 
first look at studies that used tests and then at 
studies that measured learners’ perception of gain 
through self-report. 

1973 National Evaluation 

The 1973 evaluation of the federal adult literacy 



Criteria for Selecting Studies 

• The study included an outcome/impact 
component. 

• The report was adequately documented in 
respect to design and methods. 

• There were an adequate number of cases. 

• The sampling plan was adequate ( i.e., could 
and did result in external validity). 

• Data collection procedures were adequate, 
(i.e., were not tainted by substantial attrition or 
biased by other factors). 

• Objective measures, rather than self-report, 
were used to measure outcomes. 

• Measures, especially tests, were valid and 
reliable. 

• The research design included a control group. 

• Inferences logically followed from the design 
and data. 



education program was contracted to the System 
Development Corporation (Kent, 1973). It began 
in 1971 and ended two years later. At that time, 
the Adult Education Act restricted service to 
adults at the pre-secondary level, so the study was 
limited to learners with fewer than nine years of 
schooling. The study also excluded English for 
Speakers of Other Languages (ESOL) and learners 
older than 44. For the sample, states were selected 
according to a strati ed random sampling design 
and programs and learners were selected using 
other methods of random sampling. 

After reviewing learning gain tests available in 
1972, the System Development Corporation 
selected two tests from level M of the Test of 
Adult Basic Education (TABE) to use as their 
instrument. One measured reading 
comprehension, the other measured arithmetic. 
The validity and reliability of the TABE 
components used were not reported. After 
developing directions and field testing, tests and 
instructions were sent to local program directors 



1 .34 PARTICIPANTS’ MATERIALS FOR HOW ARE WE DOING? - NCSALL TEACHING AND TRAINING MATERIALS 





Page 3 of 9 



and teachers in the study sample, who were asked 
to do the testing. The learning gain test was first 
administered in May, 1972, and then again the 
following May. Of the 1,108 initial tests obtained, 
matching tests from the first and second 
administration were obtained for only 441 
subjects. Strictly speaking, the tests administered 
were not pre- and post-tests, since at initial testing 
learners had already received varying degrees of 
instruction. 

When initially tested with components of the 
TABE, on average, learners scored at grade level 
5.4 on reading achievement and 6.4 in 
mathematics. Raw scores were not reported. After 
the second administration of the test 
approximately four months later, in which a 
different test form was used, 26 percent of the 
students had gained one grade or more in reading, 
41 percent had some gain, but less than one grade, 
and 33 percent had zero or negative gain. In 
mathematics, 19 percent gained one or more 
grades, 46 percent gained some, but less than one 
grade, and 35 percent showed zero or negative 
gain. 

The proportions of those who gained and those 
who did not may have been affected by the 
differing hours of instruction learners had amassed 
between first and second test administration. 
While almost a fifth of the learners had 39 or 
fewer hours of instruction between the first and 
second testing, another fifth had 80 or more hours 
of instruction. Average gains for reading were 0.5 
grades after 98 hours and 0.4 grades after 66 hours. 
For mathematics, the comparable figures were 0.3 
grades and 0.3 grades respectively. 

In the 1973 National Evaluation, we see many 
problems. Because of high attrition, the test scores 
are not representative of adult literacy learners in 
general. Furthermore, what do the gains reported 



mean? Are they high, medium, or low? In the 
absence of standards against which to assess 
learning gain, we do not know. 

California GAIN Study 

GAIN (Greater Avenues for Independence) was 
California’s JOBS (Job Opportunities and Basic 
Skills) program. The tested learning gain data 
comes from the larger evaluation of the entire 
GAIN program conducted by the Manpower 
Development Research Corporation (Martinson 
& Friedlander, 1994). The GAIN evaluation 
included an experimental design, explained in the 
next paragraph. Between seven and 14 months 
after a county implemented GAIN, those welfare 
recipients who had scored below 215 on the 
Comprehensive Adult Student Assessment 
System test (CASAS) were randomly assigned 
either to a treatment group or to a control group. 
The treatment group was required to attend 
JOBS'Sponsored adult literacy education classes; 
the control group was not required to attend 
JOBS'Sponsored classes, but could attend non- 
JOBS sponsored classes if they wished. As its 
learning gain test, GAIN used the Test of Applied 
Literacy Skills (TALS) quantitative literacy 
section, which is similar to the quantitative 
literacy test used by the National Adult Literacy 
Survey (NALS). The test was administered to 
1,119 treatment and control group members in 
their homes two to three years after random 
assignment. During that period learners had 
received an average of 25 1 scheduled hours of 
instruction. 

The experimental design of GAIN is very 
important for the credibility of the research. In an 
experimental design, subjects are randomly 
assigned either to a treatment group, which in this 
case received instruction, or to a control group, 
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which does not receive the treatment. Because 
random assignment insures that the two groups are 
same in every aspect except the treatment, when 
the performance of the two groups is compared, 
any difference between them can be logically 
attributed to the treatment. In short, an 
experimental design allows us to infer that adult 
literacy education caused an outcome to occur. 
This is critical because many outcomes, increased 
pay and welfare reduction for example, are 
susceptible to economic and social forces that 
have nothing to do with participation in adult 
literacy education. Thus, in the absence of an 
experimental design, we cannot be sure that 
participation caused the gains measured. 

The researchers found that, on average, learners 
gained a statistically non-significant 1 .8 points on 
the TALS test, a gain that was far too small to 
infer impact. However, despite the very small 
average learning gains, differences among the six 
counties were substantial. In fact, while in two 
counties the control group actually outperformed 
the treatment group, in one county the treatment 
group outperformed the control group by a highly 
statistically significant 33.8 percent. 

National Evenstart Evaluation 

Evenstart is the national, federally-funded family 
literacy program. To be eligible for Evenstart 
funding, a program must have an adult literacy 
education program, early childhood education, 
parent education, and home-based services. The 
National Evaluation was contracted to Abt 
Associates (St. Pierre et.al, 1993; St Pierre et. al., 
1995). As with the GAIN evaluation, the tested 
learning gain component of the Evenstart program 
was part of a larger study that assessed all the 
components of Evenstart. 



Learning gain was measured in two components, 

1) the National Evaluation Information System 
(NEIS), a data set of descriptive information 
collected from local programs, and 2) an in-depth 
study of ten local programs. For the NEIS, data 
were collected from families at entry, at the end of 
each year and at exit. The CASAS was used to 
test learning gain. For the in-depth study, data 
were collected from participants in ten programs 
selected because of geographic location, level of 
program implementation, and willingness to 
cooperate. The in-depth study included an 
experimental design. CASAS tests were 
administered to 98 control and 101 treatment 
group members who were adult literacy education 
participants from five of the ten programs. 
Although control group members were not 
participants in adult literacy education at the time 
of the pre-test, they were not precluded from 
future participation if they wished. Study subjects 
were pre-tested in the fall of 1991 and then post- 
tested twice, nine months later and 18 months 
later. 

For the in-depth study, valid pre- and post-tests 
were received from 64 participants and 53 control 
group members. Note two problems: this is a small 
number of subjects and there was substantial 
attrition from both groups. At the second post-test 
(18 months), a statistically non-significant 
difference of 3.7 points on the CASAS was found 
between the gains of the two groups leading the 
evaluators to conclude that Evenstart adult 
literacy instruction had not produced learning 
gain, at least in respect to the in-depth study. The 
NEIS component, which did not use an 
experimental design, did show small but 
statistically significant gains of 4.6 points on the 
CASAS after 70 hours of instruction. 
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The NEAEP 

The National Evaluation of Adult Education 
Programs (NEAEP), which was conducted by 
Developmental Associates Inc., began in 1990, 
was concluded in 1994, and issued five reports of 
findings: Development Associates (1992), 
Development Associates (1993), Development 
Associates (1994), Young, Fitzgerald and Morgan 
(1994a), and Young, Fitzgerald and Morgan 
(1994b). Costing almost three million dollars, the 
NEAEP was the largest and most comprehensive 
of three national evaluations of the federal adult 
literacy education program. Data on learners were 
collected at several points. Client Intake Record 
A was completed for each sampled student at the 
time of intake, and with this instrument, data for 
22,548 learners were collected from a sample of 
116 local programs. The sample was drawn using a 
statistical weighting system designed to enable the 
researchers to generalize findings from the sample 
to the United Sates as a whole. Client Intake 
Record B was completed for all learners who 
supposedly completed Intake Record A and 
completed at least one class. For this data 
collection, records were gathered for 13,845 
learners in 108 programs. Learner attrition from 
the study was clearly evident between the 
administration of the two instruments. Indeed, by 
the second data collection (Intake B) eight 
programs and more than 8,000 learners had 
dropped from the study: many learners who 
attended intake sessions never attended a class 
and also, some records were not forwarded to the 
researchers. After the second data collection, 
additional data were collected at five to eight 
week intervals for 18 weeks. 

The NEAEP lacked the resources to send trained 
test administrators into the field to administer 
tests, so it had to rely on program staff to give the 



tests and it had to use the tests that programs 
normally used. Because the Comprehensive Adult 
Student Assessment System test (CASAS) and 
the TABE (Test of Adult Basic Education) were in 
sufficiently wide use, they were chosen as the tests 
for the project. The programs selected for the 
study were supposed to administer either of these 
tests near the inception of instruction and again 
after 70 and 140 hours of instruction. Pre-tests 
were obtained from 8,581 learners in 88 programs 
and post-tests were received from 1,919 learners in 
65 programs. As one can see, the attrition 
between pre- and post-tests was substantial due to 
learner drop out and the failure of programs to 
either post-test or to submit the test data. 
Moreover, when Developmental Associates 
checked the tests, much of the data was so suspect 
that it was deleted from the study. The NEAEP 
was left with only 614 usable pre- and post-test 
scores, less than 20 percent of the intended 
number of valid cases. 

Based on these 614 cases, the NEAEP reported 
that ABE students received a mean of 84 hours of 
instruction between pre- and post-tests and 
attended for an average of 15 weeks. On average, 
their gain was 15 points on the TABE. Adult 
secondary students received a mean of 63 hours of 
instruction and gained seven points on the TABE. 
All gains were statistically significant at the .001 
level (Young et al, 1994a). 

The NEAEP found that learners do gain in basic 
skills, but how credible are the findings? In a 
reanalysis of the Development Associates data, 
Cohen, Garet, and Condelli (1996) concluded, 
“The implementation of the test plan was also 
poor, and this data should not be used to assess 
the capabilities of clients at intake. Some of the 
key evidence supporting this conclusion includes: 
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Only half the clients were pretested, and sites that 
pretested differed from sites that did not. At sites 
that pretested only some of their clients, pretested 
clients differed from those who were not pretested. 

Programs reported perfect exam scores for a 
substantial proportion of pretested clients. Less 
than 20 percent of eligible clients received a 
matched pretest and posttest. 

Among clients eligible to be posttested, significant 
differences exist among those who were and were 
not posttested. 

The available matched pre- and posttests were 
concentrated in a very few programs. 

These facts render the test data unusable. 
Therefore this reanalysis invalidates all of the 
findings concerning test results from the original 
analysis” (Cohen, Garet, and Condelli, 1996. 
p. xi). 

Stated simply, because of the problems noted by 
Cohen, Garet, and Condelli, the test scores 
received by the NEAEP, as with those of the 1973 
National Evaluation, are most certainly biased 
and, therefore, not representative of adult literacy 
learners in general. Perhaps, for example, the 
learners from whom valid pre- and post-test scores 
were obtained were more motivated and able and 
the scores are inflated. Perhaps they were less able. 
We simply do not know. Again, lacking standards, 
we do not know whether the gains reported 
should be considered high, medium or low. 

The Answer? 

The two national evaluations and the NEIS 
component of the National Evenstart Evaluation 
do show tested learning gain, but learner attrition 



from both national evaluations was so severe that 
we cannot generalize the results. In addition, these 
studies did not use an experimental design. In 
contrast, the two studies which did use an 
experimental design, the GAIN study and the in- 
depth component of the Evenstart study, showed 
no significant tested learning gain. Both studies 
were limited in other ways that space does not 
permit us to describe here. The studies included in 
the Outcomes and Impact Study that are not 
reported here show a similar pattern of confusion 
on tested learning gain. As measured by tests, do 
learners gain basic skills as a consequence of their 
participation? The jury is still out. 

Learners’ Perceptions 

As noted at the outset of this article, when 
learners are asked whether they have gained skills 
in reading, writing, and math as a result of 
participation in ABE, they tend to respond in the 
affirmative. The National Evaluation of Adult 
Education Programs (NEAEP) conducted a 
telephone survey of 5,401 former ABE learners; 
respondents were asked if they had gained in basic 
skills. Although many of the former learners who 
were supposed to be interviewed could not be 
found, and although many respondents had 
received very little actual instruction, 50% of the 
ABE learners and 45% of the adult secondary 
education (ASE) learners said that participation 
had helped their reading “a lot.” For math, the 
figures were 51% for ABE and 49% for ASE. 

In another national evaluation of the Federal 
Adult Education Program conducted in 1980 
(Young et. al., 1980), data were collected from 
110 local programs strati ed according to type of 
funding agency and program size. Learners were 
interviewed over the phone. Although the 
response rate to the interviews was low, 75 
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percent of those interviewed responded that they 
had improved in reading, 66 percent said they 
improved in writing, and 69 percent reported that 
they had improved in math. 

In a study in New Jersey (Darken wald and 
Valentine, 1984), a random sample of 294 learners 
who had been enrolled for seven to eight months 
was interviewed. Of the respondents, 89 percent 
said that participation in ABE had helped them 
become better readers, 63 percent reported that 
ABE classes had helped their writing, and 85 
percent said participation had helped their math. 
A study in Maryland (Walker, Ewart & Whaples, 
1981) interviewed 120 ABE learners who enrolled 
in Maryland programs and volunteered for the 
study. Of the respondents, 81 percent reported 
they could read better because of participating and 
90 percent reported their computational skills had 
improved. 

A study in Ohio (Boggs, Buss & Yarnell, 1979) 
followed up on learners who had terminated the 
program three years earlier. Data were collected by 
telephone. Of the 351 valid respondents, 96 
percent of those who said improving their reading 
was a goal reported they had reached the goal. For 
those who had math as a goal, the figure was 97 
percent. Finally, in a study in Wisconsin (Becker, 
Wesselius, & Fallon, 1976) that assessed the 
outcomes of the Gateway Technical Institute, a 
comprehensive adult literacy education program 
that operated learning centers in a wide range of 
locations, data were collected from a random 
sample of former learners who were divided into 
four categories based on the amount of instruction 
they had received. A total of 593 learners were 
contacted and asked if they would participate; 270 
usable interviews resulted. That the program 
helped them with reading was reported by 90 



percent, 83 percent reported that the program had 
helped with writing, and 82 percent reported that 
they had been helped with math. 

The Answer? 

The limitations of self-report is the problem that 
surfaces in these studies. Participation in adult 
literacy education is hard work and becoming 
literate is socially acceptable behavior. It could be 
that self-reported perceptions of basic skills gain 
are inflated by the normal human tendency to 
answer with socially acceptable responses and a 
reluctance to say unfavorable things in a program 
evaluation. In most of the studies, a large 
discrepancy existed between the number of 
learners the evaluation planned to interview and 
the number who actually completed interviews. It 
could be that those who were biased in favor of 
the evaluated programs were more likely to 
respond to interviews than those who were 
unfavorably disposed: the “if you can’t say 
anything good, don’t say anything at all” 
syndrome. Indeed, for the Maryland and 
Wisconsin studies, the respondents had 
volunteered to be included and may have been 
favorably biased in comparison to those who did 
not volunteer. 

Then again, perhaps the self-report data is 
accurate and learners are recognizing important 
gains in themselves that are too small to be 
measured by tests. Shirley Brice Heath (1983), for 
example, chronicles how being able to write for 
the first time a simple list or a note to one’s 
children is perceived as a significant benefit to 
those with limited literacy skills. It is doubtful 
that any of the tests in common usage are 
sensitive enough to register such gains. 
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Lessons 

Although the studies reviewed here are just a 
sample of those analyzed in the full report of the 
Outcomes and Impact Study, they provide many 
lessons. First, even the best outcome studies are 
limited in many ways, and these limitations 
influence findings. The most common limitation 
is a unacceptably large attrition of subjects 
between pre- and post-testing. The subjects for 
whom both pre-and post test data are available 
almost always differ substantially from those who 
were only pre-tested because those who are not 
post-tested include a high proportion of dropouts, 
then the results are almost certainly biased. 

A second common limitation is post-testing 
before substantial learning gain can be reasonably 
expected. Although what constitutes a reasonable 
time is open to debate, surely 30 hours of 
instruction is suspect and even 60 hours is 
questionable. Giving inappropriate levels of tests 
often creates ceiling or floor effects. In any test, 
there is a chance factor. When a test is too hard, 
learners score at the bottom, or floor. Since the 
scores cannot go down any further “by chance,” 
and can only go up, the chance factor artificially 
inflates the post-score. When the test is too easy, 
the opposite or ceiling effect occurs. Many of the 
tests Development Associates had to delete for the 
study in the NEAEP suffered from ceiling or floor 
effects. 

The most serious problem with testing may lie in 
the tests themselves. To be valid, tests must reflect 
the content of instruction, and the extent to 
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which the TABE or CASAS reflect the 
instruction of the programs they are used to assess 
is an unanswered question. Similarly, it may be 
that the tests are not sensitive enough to register 
learning gains that adult learners consider to be 
important. 

With some exceptions, such as the California 
GAIN Study, most outcome evaluations have 
relied on local programs to collect their data, a 
practice that is common in elementary and 
secondary education research. However, adult 
literacy education is not like elementary and 
secondary education, where the learners arrive in 
September and the same learners are still 
participating in June. Many adult literacy 
programs have open enrollments, most have high 
attrition rates, and few have personnel on staff 
who are well-trained in testing or other data 
collection. These factors confound accurate record 
keeping and systematic post-testing at reasonable 
and pre-determined intervals. Moreover, many 
adults are reluctant to take tests. 

Perhaps the most important lesson for policy and 
practice is that credible outcomes and impacts 
research is expensive and requires researchers who 
are not only experts in design and methodology 
but who also understand the context of adult 
literacy education. If data are to be collected from 
programs, staff must have the capacity to test and 
to keep accurate records. This will require more 
program resources and staff development. Good 
outcome studies help demonstrate accountability 
and enable us to identify practices that work. Bad 
outcome studies simply waste money. 
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Worksheet for Reading #3 

Lessons From NCSALL’s Outcomes and Impacts Study 

As you read this article think about the following questions. Complete this worksheet 
when you have finished reading. 

1. How does Beder define or “characterize” outcomes? 



2. How does he define impacts? 



3. What outcomes does your program document? 



4. What impacts does your program document? 



5. This article focuses on learning gains, or “acquisition of skills in reading, writing, and 
mathematics,” as an outcome. How are learning gains measured and reported in your 
program? 



6. When your facilitator passed out this reading she or he assigned a section on a particular 
study. For the study you read about, what was the learning gains outcome that was found? 
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Session 5: Agenda 



1. Introduction to outcome measurement 

2. Review of Harold Beder’s article, “Lessons from NCSALL’s Outcomes 
and Impacts Study” 

3. Outcomes as indicators of success 

4. Quality of life measurement 

5. Local documentation 

6. Documentation for your program 

7. Wrap-up 

Reflections on today’s session... 
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Quality of Life Measurement 



On a continuum from 1 (the lowest) to 4 (the highest): rank how important these things are in your life, and 
(2) give an actual situation of these things in your life. “1” would always be “worse” than “4.” 



Indicators 


Importance 
to you 


Actual situation 
in your life 


Questions on individual quality of life 






Having the place you live in as clean and as comfortable as a local hospital 






Feeling excited or interested in something on a regular basis 






Feeling connected to and understood by other people 






Having a choice of a job with wages and benefits you need and with an 
opportunity to do what you are interested in 






Not quarreling with the people living with you 






Spending more time at home/with your family 






Learning new things at school or elsewhere 






Getting dental services 






Eating meals together with your family 






Not having to go to school/work if you do not want to 






Having enough energy to do things you have to do 






Having more time and energy to do things you enjoy doing 






Getting satisfactory goods n food, appliances, clothes 






Feeling that someone cares about you 






Feeling free from threat and unexpected occurrences 












Questions on community quality of life — apply them to your community 






1. Spills and other environmental accidents promptly reported 






2. Public housing waiting lists are tolerable 






3. New cancer cases/deaths are scarce 






4. Number of violent crimes in the area is small 






5. Non-white people arrested as a certain % of all arrests 






6. A lot of scenic land within 30-mile radius 






7. School pupil/teacher ration in public schools is less than 20/1 






8. High percentage of population walking to work 






9. Numerous social/youth clubs 






1 1 . Female/male earnings ratio is 1 






12. Rate of vacation travelers is high 






13. Many high school graduates go to college 






14. Many persons report interest in politics 






1 5. There is no trash in the streets 
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Do/Set/Met Goal Sheet 

Instructor Class Date 



Please fill in the total number of students in your class that have checked DO/SET/MET in each of the 
following categories. 





DO 


SET 


MET 


COMMENTS 


1 . Fill out a job application 










2. Arrive at appointments on time 










3. Fill out work related forms 










w 4. Read and understand work related materials 










§ 5. Communicate with others in work/social settings 










® 6. Plave a job 










7. Obtain job promotion 










8. Organize, plan and prioritize work 










9. Use a computer 










10. Use library 










cc 11. Locate and/or use community agencies or services 










12. Plave a driver’s license 

Cl 










§ 13. Use public transportation 










> 14. Complete U.S. citizenship class 










| 15. Apply for legal immigrant status, U.S. citizenship, Emancipation papers 










| 1 6. Register to vote 










o 1 7. Vote in primaries/elections 










z 18. Obtain legal advice 










£j 19. Use maps 










o 20. Ask for directions 










= 21 . Active in volunteer work 










22. Participate in neighborhood watch/activities 










ffj 23. Read to children 










<| 24. Help children with homework 










g 25. Volunteer in child’s school 










h 26. Talk with teachers/school staff 










§ 27. Attend school-related meetings (ex. PTA) 










k 28. Fill out insurance forms 










w 29. Use a recipe 










£ 30. Assemble a toy/equipment/furniture 










= 31. Read product/medicine label, directions and safety warnings 










32. Read daily (magazines, books, newspaper) 










33. Have a checking/savings account 










34. Balance a check book 










35. Have a personal/family budget 










36. Pay bills 










qJ 37. Use an ATM card 










^ 38. Count money 










39. Compare prices to determine the best buys 










40. Solve a problem 










41. Feel good/better about myself 










42. Feel independent 










43. Meet a personal goal 











Developed by Rita Roper, Jerry Musick, and Sherri Whitlock, teachers in the 
Mount Rogers Regional Adult Education Program in Abingdon, VA 
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Reading #4 

A Story of Improvement 

— by Jane Cody, James Ford, and Kathleen Hayward 



An honest look told this literacy program they 
were not as good as they might be. It was time 
to listen to stakeholders and institute a 
continuous improvement process. 

“The art of progress is to preserve order amid 
change and to preserve change amid order.” 

— Alfred North Whitehead 

Looking in the mirror, really taking a long, hard look, 
can be unnerving, even depressing. We see the 
blemishes that we hide with makeup. We see the bald 
spots that we cover by creatively combing our hair. We 
see the ten extra pounds that age has brought. Most of 
us avoid the problem by not looking, or we take a 
quick glance to make sure we are properly buttoned 
and zipped and ready to appear in public. Perhaps we 
make a silent resolution about dieting or exercising, 
but that’s about it. After all, we’re not perfect. 

This is pretty normal human behavior and it 
carries over into our professional lives. After 
operating an adult literacy program for many 
years, it is easy to avoid taking a long, hard look 
in the mirror. Students come, money comes, 
programs expand, people say “well done,” 
everything seems pretty good: not perfect, but 
pretty good. 

This is the story of an adult literacy program 
that decided that merely glancing in the mirror 
was no longer enough. It is a story of shock: of 



discovering a lack of clearly defined standards and 
the tools to measure progress toward them. It is a 
story of hope, in finding a process to identify 
program gaps and narrow them. It is also a story of 
improvement, through the systematic use of 
stakeholder input, hard data, and detailed analysis. 
It is a story, not about perfection, but about 
getting better. 

Our Organization 

Adult and family literacy programs in Knox 
County, Tennessee, are provided by a unique 
public/private partnership. The public member is 
the Knox County School System and, specifically, 
its Adult Basic Education Department (ABED). 
The private member is a nonprofit corporation 
called Friends of Literacy, Inc. (FOL), which was 
formed in 1991 to supplement — by way of money 
and volunteers — the resources that ABED had for 
literacy programs. Through the balance of this 
article, “we” refers to this FOL/ABED Partnership, 
to the program it produces, and to its combined 
leadership. 

Most knowledgeable observers would have 
described this as a successful partnership with 
high'Caliber programs. Funding grew each year, 
students provided ample anecdotal evidence that 
we made a difference in their lives, and we had 
achieved some national recognition. We 
considered ourselves anything but static. We had 
changed our curriculum, improved our volunteer 



Article reprinted from Focus on Basics, Vol. 2, Issue C, September 1998 
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training, and added more paid teachers. Our 
program had grown to approximately 400 
students, 15 paid staff, and 40 volunteers. We 
were doing something right. 

Forces for Change 

In 1996, FOL had its fifth birthday. Its budget had 
grown from $5,000 in 1991 to more than 
$150,000 in fiscal year ’96-’97. It had evolved 
from an organization that provided money and 
people to an adult literacy program operated by 
ABED to one that was also the primary 
management and fiscal agent for its own family 
literacy program. It was time to take a look at 
where FOL, the Adult Literacy Program, and the 
partnership were going. As luck would have it, 
new members had just been added to the FOL 
board of directors. These members had not grown 
up with the organization and, consequently, 
viewed it from a different perspective. One new 
member had expertise in strategic planning. A 
committee composed of FOL board members and 
program staff reviewed and clarified the vision, 
mission, and values of FOL. 

In addition, the FOL/ ABED partnership had 
been selected to participate in a national project 
entitled “What Works Literacy Partnership” 
(WWLP). WWLP sparked our thinking, in a 
more detailed way, about how we could document 
the results we thought we were achieving in 
improving the basic academic skills of our 
students. We started to concentrate on how 
could we prove that our program really did what 
we said it did. 

Dovetailing with WWLP was another national 
initiative entitled “Equipped for the Future” 
(EFF).The FOL/ ABED partnership was selected to 
be one of several pilot-test sites for EFF. Like 



WWLP, one aspect of EFF focused on the question 
of how we measure results. 

Two other critical pieces fell into place during the 
summer of 1997. First, to help us with the WWLP 
project, we hired a new staff member who just 
happened to be a Certified Quality Examiner for 
the State of Tennessee (more about this below). 
Second, FOL received a generous gift to purchase 
new hardware and a software system to manage 
data about students, volunteers, and donors. With 
all this going on, we had both the courage to look 
in the mirror and some ideas about how to take a 
systematic approach to dealing with what we saw 
there. 

New Approach 

Our new staff member introduced us to the 
Malcolm Baldrige Educational Criteria for 
Performance Excellence. These criteria take the 
total quality management principles originally 
used in the private sector and apply them to 
educational institutions. They provide an orderly 
approach to continually improving the operations 
and results of an educational enterprise. This was 
the systematic approach we would use. 

The starting point was to identify the needs of the 
customers: in this case, our students. FOL/ABED 
program leadership instituted monthly “Town 
Meetings” to ask students what they wanted, 
needed, and expected from our adult literacy 
program. Approximately 75 students attended 
each of the meetings. We got a wide range of 
responses, from better parking to more computer 
time and more volunteers for individual tutoring. 
Next, we asked teachers and volunteers, who are 
key stakeholders in the program, what they 
expected and how we could all work together to 
improve things. Better training and more support 
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from the leadership were two answers. This group 
also developed its own vision, mission, and value 
statements for both adult and family literacy 
program components. These built on similar work 
completed earlier by FOL and the Knox County 
Schools, but were the work of these stakeholders, 
and were something they could commit to as their 
own. The FOL board also brainstormed its role 
and the expectations of and for each board 
member. Some consistent themes were emerging, 
one of which was better communication among 
the stakeholder groups: students, teachers, 
volunteers, and board members. 

We wanted to use this information to develop 
quality standards for our organization and 
program. We were fortunate: adult literacy 
educators in both Tennessee and New York had 
already adopted “Indicators of Program Quality” 
that fit nicely into the Baldrige Criteria and our 
own vision, mission, and values. For example, 
Quality Indicators One and Two of the New York 
program stress the involvement of all stakeholders 
in the development of program philosophy and 
direction. Quality Indicators Three and Five 
underscore the importance of measuring student 
improvements in reading, writing, speaking, and 
problem solving, and highlight the goal of 
enabling students to be better workers, parents, 
and citizens. These Indicators were consistent 
with our approach in the EFF project. Indicators 
Seven and Eight set benchmarks for the 
recruitment, orientation, placement, and 
retention of students. We felt comfortable with 
all 13 Quality Indicators and adopted them as 
standards by which to measure our own work. 

Measuring progress towards benchmarks and 
ultimate program results are critical concepts in 
the Baldrige Criteria. We realized that we must 
start “managing by facts,” rather than by 



assumptions or guesswork. We set about the task 
of collecting and analyzing data in several ways. 
First, program management went back into the 
classroom to see what actually happened there on 
a daily basis. They resumed the role of teacher to 
see, firsthand, the problems of taking our 
curriculum from training manual to actual 
delivery. They also observed other teachers to 
determine whether the material covered during 
in-service training was actually being 
incorporated. These were eye-opening 
experiences. What we thought was happening in 
the classroom was not always occurring. Some of 
the curriculum was unwieldy or too complex. It 
was not always well integrated between teachers 
in different classrooms. In our family literacy 
program, for example, the curriculum used by 
those teaching the parents was not well linked 
with that used to teach the children. Teachers 
provided input to help redesign curriculum areas 
that were not working. 

We needed better data on actual student 
performance, so we changed to a more accurate 
instrument that provides diagnostic information 
to the student and the teacher. We tested adult 
learners using the Test of Adult Basic Education 
(TABE) when they first enrolled in our program, 
and again after completing 100 hours of classroom 
instruction. We collected, analyzed, and fed data 
back to both teachers and students. We use this 
information to develop specific, individualized 
action plans for improvement by each adult 
learner. At last we had reliable, hard data to show 
us whether our students were making academic 
progress. 

As we were gathering data, a “think tank” of 
program managers, teachers, and FOL board 
members began to identify the “vital few” core 
drivers of our partnership and literacy programs. 
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Core drivers are what we have to do well to meet 
the indicators of a quality program. They included 
student recruitment and retention, volunteer 
management, financial management (raising and 
spending money), and bringing our new data 
system up to speed. The group examined how we 
operated in each of these areas, set sights on a 
higher standard for future performance, and 
developed detailed action plans for improvement. 
For example, we knew our data system was critical 
to managing by fact. Without a better system, we 
could not record and analyze data on student test 
results. Neither could we keep track of who had 
given us money, when, how much, and for what 
purpose. We solicited input from everyone who 
had to use the data system, asking, “What are the 
problems with current operations and how could 
they be improved?” We selected a point person to 
deal with the software company in redesigning 
some elements of the system, based upon this 
input, and we established a timetable for 
improvement. 

Shock, Hope, Improvement 

The Baldrige criteria require an organization 
to slow down and rebuild with a focus towards 
customer and program results that matter. The 
criteria link and create an alignment between all 
internal operations, organization leadership, 
systems, and processes as well as sound literacy 
standards. When we first began our critical look, 
many of us confidently believed that, on a scale of 
one to ten, with ten being the best, we would be 
about an eight. Wrong. Systematic feedback, data 
collection, and analysis, as described above, 
showed some real gaps in our operations. For 
example, teachers told us that the continuous 
enrollment of new students on whatever day they 
happened to show up was a real barrier to success. 
It required teachers to interrupt classes to assess 



new students and try to bring them up to speed. 
Students told us that it was intimidating to be 
assigned to a classroom without any idea of what 
to expect upon arrival. The solution, devised with 
input from the stakeholders, was a new class, 
called Learning Skills, in which each new student 
is enrolled for a month before assignment to a 
regular class. Our TABE test results indicate that 
students who attended the Learning Skills class 
are progressing more quickly than those who were 
enrolled in our program prior to development of 
this class. In addition, since students graduate 
from Learning Skills at the end of the month, 
teachers in regular classes enroll new students only 
once a month and have benchmarks for them 
before they arrive. 

Teacher training was another area that needed to 
be addressed. We found that teachers were not 
incorporating in-service material in their 
classrooms. We asked them for help in revising 
classroom curriculum, and we changed our 
training focus from putting on a workshop’ to 
meeting the needs of the staff. In addition, our 
EFF implementation allowed us to bring together 
teachers who worked in different parts of our 
program. Lack of funding for teacher planning 
time had been a significant barrier to this in the 
past. Adult literacy teachers linked up with family 
literacy teachers. Staff began exchanging ideas 
and method; they brainstormed to solve problems. 
When asked to evaluate this year upon its 
completion in June, 1998, teachers were 
significantly more positive about the program, and 
their role in it, than they had been in the past. 

We also saw another significant change during the 
year. We had tried several times in the past to 
develop a leadership group from within the 
student body. Efforts had always failed. This year, 
they blossomed. Several students volunteered to 
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serve on a Leadership Council that tackled some 
successful projects. Why the difference? The 
projects were those that mattered to the students. 
Their input drove the leadership group’s agenda. 
The students on the Leadership Council took data 
from the Town Meetings and also surveyed 
students about project priorities, using the 
information to chose items to tackle first. The 
initial project created a student eating area. 
Members of the Council used a quality 
improvement tool to plan and then establish this 
new space. They also described their process and 
results at an FOL Board meeting. The next 
project, still underway, is recycling aluminum cans 
to raise money to open an on-site store where 
students can buy paper, pens, and other school 
supplies. The evidence is anecdotal but, 
nevertheless, important: Students exhibited a 
greater sense of community and program 
ownership than in past years. While student- 
centered’ has always been important to our 
partnership, we found a new way to put it into 
practice. 

Our testing program produced data showing that 
our students are improving their reading, writing, 
spelling, and math skills. Preliminary results 
indicate that students gained an average of one 
grade level in each of these areas after 100 hours 
of classroom instruction. These data have helped 
us to identify strong teaching techniques and 
discard the less effective. The information has 
been heartening to students and teachers, who 
have objective evidence that their work is paying 
off. It has also been helpful in fundraising, since 
we can show prospective donors that investing in 
our programs will get results. We have more 
credibility with these data, and are more 
competitive. 



Reflections 

There is an old saying: “It’s more important to be 
lucky than good.” We knew we had been lucky, 
and we thought we had been good. But we wanted 
to be better. We decided that good intentions, 
luck, and ad hoc methods had taken us as far as 
we could go. We could no longer excuse our 
shortcomings by saying that ours was a fluid’ or 
dynamic’ environment, or one that was 
underfunded, implying that somehow our 
expectations should be lower as a result. It was 
time for accountability and documented results 
through systematic program improvement. 

This has been a re-education for all of us. The 
vocabulary, which comes from the business sector, 
is new to us. The process can also be discouraging, 
especially at the beginning when scrutiny revealed 
some things we would rather not have seen. 
Indeed, when our Think Tank sat down to 
identify the “vital few” areas to address rst, we 
came up with a list of “vital many” and had 
difficulty prioritizing where to start. 

Commitment 

This process requires the belief that it will work as 
well as the commitment of time to think, plan, 
and involve students, staff, board members, and 
volunteers. All of these groups have been 
involved and supportive. The Baldrige Framework 
is a very logical approach, built on common sense. 
For those of us geared toward instant results, it has 
sometimes been frustrating to slog through the 
tedious process of documenting, in detail, the way 
we do things (our “as is” state) versus the way we 
should do things (our desired state). It is time 
consuming and requires discipline. Just recently, 
three staff and one board member spent an hour 
discussing and writing down the process we will 
use to handle receipts and expenditure. Now we 
have a simple procedure that everyone involved 
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has agreed upon. It replaces a hit-and-miss 
method that no one understood. The people 
involved had to learn process charting and 
develop this particular chart in addition to their 
other duties, not in place of them. If the 
investment of time produces a better process that, 
in turn, produces better results in this instance, 
fewer mistakes, less confusion and duplication of 
effort, more reliable financial data then it will 
save future time and effort. We do not worry 
about getting so wrapped up in process charting 
that we forget the ultimate goal is to achieve 
results in student performance, because student 
feedback and student testing helps keep us 
focused. 

Our world is more complex now because we 
understand how the various parts of our program 
are interrelated. All our staff and volunteers need 
to be open to change and to soliciting and 
receiving feedback. If we have used customer 
input to design the process, and measured and 
analyzed both customer satisfaction and results 
along the way, we should arrive at our goal. If we 
do not, we have a good idea of what went wrong 
so that we can improve the next time around. We 
now understand that orderly is not synonymous 
with bureaucratic. 

We have a long way to go. In the coming year, we 
will be working hard to bring the ABED and FOL 
leadership further on board. Although both have 
been supportive, both groups need a better 
understanding of the significant organizational 
implications of this continuous improvement 
process. We will be implementing more fully our 



EFF curriculum. We will continue work on our 
action plans for the “vital few” and document our 
progress toward improvement. 

Deciding to improve the performance of an 
organization in an orderly, comprehensive, 
systematic way is analogous to an individual 
deciding to lose the ten pounds that a stop in 
front of the mirror revealed. It entails hard work, 
discipline, incremental results, motivation 
through small victories and, most importantly, a 
change in lifestyle. Just as a fad diet does not 
produce long-term success in weight reduction, fad 
quality’ is not the answer for improving 
performance over time. An organization must live 
and breathe quality and incorporate it as a cultural 
value in everything that it does. We have only 
begun that process, but, as the saying goes, “a 
journey of a thousand miles begins with a single 
step.” 
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1. Introduction to final session 

2. Review of “A Story of Improvement” 

3. Identifying questions about program performance 

4. Approaches to improving program performance 

5. Next steps for your program 

Reflections on today’s session... 
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Reading #5 

Some Approaches to Improving Program Performance 

— by Mary Beth Bingman, written for this guide 



Improving program performance can be addressed 
using a variety of approaches n or a combination 
of approaches as in Knox County. In deciding 
what approach to take, programs need to consider 
time available, support available, and how an 
approach matches their style and values. The 
following three approaches are all being used 
nationally, but level of support for each vary from 
state to state. Any approach to improving program 
quality will only work if a program can commit 
time and focus to the process. 

Program Quality Standards 

For many years the federal and state adult 
education systems have promoted the use of 
program standards. States and other groups have 
developed Program Quality Indicators that 
programs can use to evaluate the various 
components of their program and to plan change. 
In this reading we have included information on 
program standards for ESOL programs recently 
developed by TESOL (Teachers of English for 
Speakers of Other Languages). Their Quality 
Indicators describe what TESOL considers a 
quality ESOL program. The TESOL Program 
Standards do not describe the process for 
improving practice, but do provide a list of 
indicators of a quality program as developed by 
the TESOL task force. Even Start has developed 
a list of program quality attributes for family 
literacy program, and many states have indicators 
of program quality. To use these quality indicators 
a program might make an indicator-by-indicator 
assessment of “how they are doing” and then plan 
for the changes that would improve performance 
on particular indicators. 

Content Standards 

While the TESOL indicators are for program 
quality, Equipped for the Future (EFF) is based on 
content standards. Content standards address 



what is taught and learned. Many states have 
developed content standards for subjects in the K- 
12 curriculum. The sixteen EFF content standards 
are part of a framework developed through a 
process involving students, practitioners, and 
thousands of stakeholders in the adult literacy and 
lifelong learning system. The framework identifies 
adults’ purposes for literacy and what adults need 
to know to carry out their responsibilities as 
workers, family members, and community 
members. The EFF content standards focus on the 
application of skills in adults’ lives. This reading 
includes a brief introduction to EFF, the 
“standards wheel” that identifies the sixteen 
standards, and an example of one standard. The 
EFF initiative of the National Institute for 
Literacy is developing as assessment framework to 
accompany the standards. The EFF framework and 
standards do not provide a specific set of 
indicators in the same way that program standards 
do. But the framework does provide a variety of 
tools including a guide to EFF approach to quality 
that programs can use to examine their practices 
and to plan and implement changes. 

Continuous Improvement 

The Malcolm Baldrige Educational Criteria for 
Performance Excellence are based on total quality 
management principles originally developed in 
business sector. The Baldrige criteria outline an 
orderly process for examining and improving 
program operations. They provide the questions to 
ask but not the answers. Where program and 
content standards define quality, the Baldrige 
criteria focus on processes to build quality. This 
reading includes a graphic presentation of the 
process and a review of the key items in each 
category. 

Materials from these three approaches follow. 
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TESOL Program Standards 

The program quality standards developed by The Teachers of English to Speakers of Other 

Languages (TESOL Task Force on Adult Education Program Standards, 2000) describe 

program quality indicators in eight areas: 

• program structure, administration, and planning; 

• curriculum; 

• instruction; 

• recruitment, intake, and orientation; 

• retention and transition; 

• assessment and learner gains: 

• staffing, professional development, and staff evaluation; 

• support services. 

Examples of indicators for the area of structure, administration, and planning include: 

1 . The program organizes its instructional offerings to be consistent with the program’s mission 
and goals and with the goals and needs of learners in the community being served (TESOL 
Task Force on Adult Education Program Standards, 2000, p. 19). 

2 . The program maintains a student teacher ratio conducive to meeting learning needs and goals 
within the constraints of program guidelines (p.20). 

3. The program has a plan for outreach, marketing, and public relations activities to foster 
awareness and understanding of the program (p. 21). 

Examples for curriculum include: 

1 . Curriculum and instructional materials are easily accessible, up'tO'date, appropriate for adult 
learners, culturally sensitive, and oriented to the language and literacy needs of the learners 
(TESOL Task Force on Adult Education Program Standards, 2000, p. 21). 

2 . The program has an ongoing process for curriculum revision in response to the changing needs 
of the learners, community, and policies (p. 22). 

Examples for instruction include: 

1 . Instructional activities engage learners in taking an active role in the learning process (TESOL 
Task Force on Adult Education Program Standards, 2000, p. 22). 

2. Instructional activities take into account the needs of multilevel groups of learners, particularly 
those with minimal literacy skills in their native language and/or English (p. 23). 

For more information about TESOL program quality standards, visit TESOL Web site 

www.tesol.org. 

From Program standards for adult education ESOL programs (2000). 

Teachers of English to Speakers of Other Languages, Inc. 
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Equipped for the Future 

— Excerpted from the EFF Web site: www.nifl.gov/links/collections/eff/eff.html. 



Equipped for the Future (EFF) is the National 
Institute for Fiteracy’s standards based system 
reform initiative aimed at improving the quality 
and outcomes of the adult literacy and lifelong 
learning delivery system. EFF starts from the 
recognition that the skills adults need as parents, 
workers, and citizens go beyond the basic 
academic skills that have traditionally been 
targeted by adult education programs. 

NIFF undertook EFF in an effort to better 
understand what we need to do as a nation to 
meet the challenge posed in the National Goal for 
adult education and lifelong learning: “Every adult 
American will be literate and possess the 
knowledge and skills necessary to compete in a 
global economy and exercise the rights and 
responsibilities of citizenship.” 

Through a grass roots process, EFF partners have 
engaged thousands of customers and stakeholders 
in the adult literacy and lifelong learning system 
to build consensus on customer needs and goals. 
From this consensus EFF Partners are developing, 
refining and validating frameworks for content 
and performance standards. 

The frameworks are a primary vehicle for enabling 
system reform that lines up instructional practice, 
program services, and program accountability so 
that they all focus on learner goals and stated 
purposes. 

Content Framework 

The foundations of the framework are the four 
purposes for literacy identified by adult learners: 

• access to information so adults can orient 



themselves in the world 

• voice — being able to express ideas and opinions 
with the confidence that they will be heard and 
taken into account 

• independent action — being able to solve 
problems and make decisions on one’s own, 
without having to rely on others 

• bridge to the future — learning how to learn, in 
order to keep up with the world as it changes 

The role maps describe what adults need to know 
and be able to do to carry out responsibilities as 
citizens/community members, parents/family 
members, and workers. The three components 
that complete the framework structure — the 
activities common to all three roles, the standards, 
and the core knowledge domains — are all linked 
to the four purposes, and through the role maps, 
to the key contexts in which adults live. 

Standards 

The 16 EFF Content Standards are derived from 
research that helped us clarify the core knowledge 
and skills adults need to carry out their roles as 
family members, workers, and community 
members. EFF Standards focus on application of 
skills - what adults need to know to actually use 
skills effectively in daily life. 

For more information about Equipped for the 
Future framework, visit EFF Web site: 
www.nifl.gov/lincs/collections/eff/eff.html 



1 .56 PARTICIPANTS’ MATERIALS FOR HOW ARE WE DOING? - NCSALL TEACHING AND TRAINING MATERIALS 




Page 4 of 8 



EFF Content Standards 




^ R S o N A L 



From Stein, S. (2000). Equipped for the Future content standards: What adults need to 
know and be able to do in the 21st century. Washington, DC: National Institute for 
Literacy, p. 21. 
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Sample EFF Standard 



DECISION-MAKING SKILL 

Use Math to Solve Problems and Communicate 



In order to fulfill 
responsibilities as 
parents/family members, 
citizens/community 
members, and workers, 
adults must be able to: 




STANDARD 



Use Math to 
Solve Problems 
and Communicate 

• Understand, interpret, and work 
with pictures, numbers, and 
symbolic information. 

• Apply knowledge of mathematical 
concepts and procedures to figure 
out how to answer a question, 
solve a problem, make a prediction, 
or carry out a task that has a 
mathematical dimension. 

• Define and select data to be used in 
solving the problem. 

• Determine the degree of precision 
required by the situation. 

• Solve problem using appropriate 
quantitative procedures and verify 
that the results are reasonable. 

• Communicate results using a variety 
of mathematical representations, 
including graphs, chart, tables, and 
algebraic models. 





From Stein, S. (2000). Equipped for the Future content standards: 
What adults need to know and be able to do in the 21st century. 
Washington, DC: National Institute for Literacy, p. 35. 
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Using Baldrige Criteria as a Process 




Credit: Jim Ford 



Customer Requirements 



As leader I will 



My Action Plan is 



The people who will do this are 



The Key Process we will use will be 



We will measure our performance by 



The bottom-line results will be 
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Baldrige Criteria Category Review and Key Items 



Category 1: Leadership 

Have a clear understanding of how your 

leadership system works. 

1.1 Organizational Leadership: 

• Mission, Vision, and Values 

• Have a clear understanding of how your 
leadership system works (Organizational Chart). 
Show how the system includes a representative or 
receives internal and external feedback 

Organizational Performance Review 

Have an understanding of how you obtain and 
receive performance feedback to know and 
measure how well the program is doing, as well 
as make adjustments and improvements. 

A list or table showing what are the feedback 
mechanisms, who provides this feedback, and how 
they are used. Have samples of the documents and 
possible results if the system is that mature. Use a 
flowchart to show the feedback system. 

1.2 Public Responsibility and Citizenship: 

• Explain how you, your staff, and program takes 
responsibility and contributes to the community 
Use a list or table showing what the program has 
done, by whom, and for what group. 

Category 2: Strategic Planning 

2.1 Strategic Development: 

• Explain your planning process long and short- 
term and how it directs your goals and plans for 
customer satisfaction and program improvement 
Use a flowchart to show how the major steps in the 
strategic planning process. 

2.2 Strategic Deployment: 

• How does your program deploy your strategies 
and action plans to insure performance to reach 



results? Using the organizational chart or another 
flowchart, show how plans are communicated and 
applied through the organization. 

Category 3: Customer (Student) 
and Market Focus 

3.1 Customer (Student) and Market 
Knowledge: 

• Who are your primary customers and what are 
their expectations and requirements? 

Use a matrix to show the different customers and 
their requirements. 

C.O.P.l.S. Worksheet. 

3.2 Customer Satisfaction and Relationship: 

• How do you measure customer satisfaction and 
build relationships? 

If you use a survey, show the results in a graphic 
form. 

Customer Satisfaction Survey 

Category 4: Information 
and Analysis 

4.1 Measurement of Organizational 
Performance: 

• What are the key information and data 
(performance measures) that you collect on the 
performance of the organization? 

Use an Information Matrix 

4.2 Analysis of Organizational Performance: 

• How do you analysis the data and information? 
Use an Informational Analysis flowchart. 
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Category 5: Human Resources 
Focus 

5.1 Work Systems: 

• How do you and your program design jobs and 
work process so employees can achieve high 
performance? Use a basic work/ job design 
flowchart. 

5.2 Employee Education, Training, and 
Development: 

• How does your program provide training and 
education to build employee knowledge, skills, 
and capabilities that improve employee 
performance? Use a matrix to list the training 
opportunities for the staff, specific subjects covered, 
and the projected impact of that training. 

Use a flowchart to show the basic steps in developing 
the staff. 

5.3 Employee Well-being and Satisfaction: 

• How do you measure or know how your workers 
feel about their work, program, and their overall 
satisfaction level? 

Staff Satisfaction Survey. 

Category 6: Process Management 

6.1 Product and Services Processes: 

• How do you manage the design and delivery of 
your key processes? 

Matrix of the key processes . 

Flowcharts of key processes. 

6.2 Support Processes: 

• How do you manage key support processes? 
Matrix of the key support processes. 

Flowcharts of key support processes . 

6.3 Supplier and Partnering Processes: 

• How do you manage key supplier and/or 
partnering interactions and processes? 



Matrix of the key supplier processes. 

Flowcharts of key supplier processes . 

Category 7: Business Results 

7.1 Customer (Student) Results: 

• What are the customer (student) results, 
including satisfaction? Use a few key graphs to 
show results and improvements. 

7.2 Financial and Market Results: 

• Do you have any financial or market place 
performance results? This could be a place where 
you show how increases in budgets, etc. 

7.3 Human Resources Results: 

• What are the employee satisfaction, well-being, 
and development results? Use a few key graphs to 
show results and improvements. 

7.4 Supplier and Partner Results: 

• What are the supplier and partner results? 

This could be a place where you show how increases 
in improved supplier services and key partnerships 
leading to better service results and performance. 
Use graphs to show results. 

7.5 Organizational Effectiveness Results: 

• What are other important operational 
performance results that contribute to the 
program’s effectiveness? In the educational criteria 
this is the place for scholarship, research, etc. Use 
graphs to show results. 

Note: Materials in this reading related to the Baldrige 

Criteria were provided by Jim Ford by the Center for 

Literacy Studies. 

For more information on Baldrige Criteria, visit the 

Web site: www.quality.nist.gov 



PARTICIPANTS’ MATERIALS FOR HOW ARE WE DOING? - NCSALL TEACHING AND TRAINING MATERIALS 1 .61 




APPENDIX 2. 



Suggestions 
for Facilitators 



HOW ARE WE DOING? - NCSALL TEACHING AND TRAINING MATERIALS 2.1 




Facilitating the Process 

— by Mary Beth Bingman, written for this Guide 



Facilitating a group requires a variety of skills. 
While the instructions in this guide are quite 
specific, you will run into unanticipated 
situations and you may want to make 
revisions to the activities. This section draws 
from the book Teacher as Learner (Bingman & 
Bell, 1995) to describe good facilitation. 
Appendix Two includes several readings from 
the Study Circles Resource Center that may 
also be helpful to you. 

Effective facilitation: 

• has shared responsibility and accountability 
among facilitators and participants for the 
outcomes of the experience 

• sets an atmosphere which respects the 
dignity and diversity of all involved. It 
breaks isolation and builds a community of 
learners 

• combines structure with freedom and 
flexibility. The role of the facilitator is to 
encourage and guide, not control, the 
process. 

Good facilitators are adept in: 

• understanding group process 

• listening 

• asking questions 

• giving and receiving feedback 

• getting everyone’s ideas 

• summarizing 



Understanding Group Process 

It is generally accepted that groups that come 
together for a particular task move through 
recognizable stages of “togetherness.” Several 
models of group developments describe five 
phases: politeness, clarification and purpose, 
power, working together, and proficiency. Being 
familiar with these phases can help facilitators 
choose appropriate activities and topics and 
be comfortable with changes in group 
behavior. A simple way to remember the five 
stages of this group development process is 
through the jingle: Form, Conform, Storm, 
Norm, Perform. 

Listening 

Listening is often described as a passive skill, 
but those who have tried to be effective 
listeners know that it takes total 
concentration and involvement. The term 
“active listening” is more than just a catch 
phrase for good human communications; it 
really does mean active involvement on the 
part of the listener. Some suggestions for 
active listening include: 

• Maintain eye contact with the speaker. 

• Nod your head or say “yes,” or “uh-huh” to 
indicate that you’re listening. 

• Use open-ended questions to encourage the 
speaker to talk more. 

• Show by your body language that you’re 
interested; lean toward the speaker, don’t 
fidget. 
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• Summarize the speaker’s comments, or say 
them back in a way that lets the speaker 
know you’ve understood. 

• Let there be silence. Don’t try to fill every 
pause. 

• Keep the focus of the conversation on the 
speaker; don’t disagree or talk about 
yourself. 

Asking Questions 

A question, especially a well-timed open- 
ended question can often move a group to a 
new level of understanding or help the group 
see where it needs to go next. Open-ended 
questions, unlike closed questions that 
usually have an exact answer, don’t have one 
right answer. Questions such as “What’s 
going on?” “Are there any surprises?” “What 
does this suggest to you?” stimulate thinking 
and build the knowledge of the group. 

Giving and Receiving Feedback 

Facilitators of groups need to be able to 
recognize feedback, know when giving 
feedback is appropriate, and be able to 
distinguish helpful and unhelpful ways of 
giving feedback. Some tips on giving and 
getting feedback include: 

• Talk in the first person 

• Be specific or probe reasons 

• Acknowledge how you connect to a 
problem 

• Wherever possible, make suggestions for 
alternative approaches 

• Don’t assume that the difference is 
political 



Getting Everyone’s Ideas 

An inquiry group is dependent on full 
participation from everyone. Facilitators set 
an expectation for everyone to be “fully 
present” and active in the learning process. 
You also need to take care to monitor your 
own “talk time.” When the facilitator is 
talking, the participants aren’t. Often a few 
participants have a great deal to offer, and 
while you want their contributions, you also 
need to be sure that you don’t put other 
participants in the position of having to 
interrupt in order to speak. 

Summarizing 

Summarizing pulls the group back into focus. 
It keeps the discussion moving forward when 
participants are beginning to ramble or get 
off track. It helps move discussion from one 
phase to another, for example from sharing 
to interpreting to generalizing to applying. It 
helps participants see and clarify their own 
knowledge and ideas. Summarizing can be 
done both verbally and graphically. The 
facilitator at key moments can say something 
like: “I hear you saying...” or “So far we have 
discussed...” or “This is what I have been 
hearing...”. Then you might follow with a 
question, “What else?” or “Have I 
understood this?” or “Do you agree on this?” 

Much of the work of the inquiry sessions 
described in this guide will not require 
intense facilitation, but in activities such as 
discussing readings or pulling together what 
has been learned from the activities, you will 
need the skills described here. 



2.4 



HOW ARE WE DOING? - NCSALL TEACHING AND TRAINING MATERIALS 




V Key facilitation skills 



• Reflecting - feeding back the content and feeing of the message 

j=t me see if I’m hearing you correctly 

• Clarifying - r estating an idea o r thought to make it more dear 

"What i believe you are saving is 



• Summarizing - stating concisely the mam thoughts 

it sounds to me as if we have been talking about a few 
major themes 



• Shifting focus - moving from one speaker or topic to another 
Thank you John Do you have anything to add, Jane 7 ’ 

"We ve been focusing on views 1 and 2 Does anyone have 
strong fee 5 mgs about the other views 7 " 



• Using silence - allowing time and space for reflection oy pausing 
be ween comments 

• Using non-verbal and verbal signals - combining body 
language and speech to communicate - for example, using eye 
contact to encourage or discourage behaviors in the group Be 
aware of cultural differences 

Neutrality is important here so that we don't encourage some 
people more than others 



©Topsfield Foundation. Reprinted with permission from The Guide for Training Study Circle Facilitators 
(1998) by the Study Circle Resource Center. 
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jy| Suggestions for dealing 
with typical chal enges 

Most study circles go smoothly because participants are there voluntarily and have 
a stake in the program. But there are challenges in any group process. What follows are 
some of the most common difficulties that study circle leaders encounter, along with 
some possible ways to deal with those difficulties. 

Problem: Possible responses: Try to draw out quiet participants, but 

Certain participants don't put them on the spot. Make eye contact - it reminds 

don’t say anything, them that you’d like to hear from them. Look for nonverbal 

seem shy. cues that indicate participants are ready to speak. Frequently, 

people will feel more comfortable in later sessions of a study 
circle program and will begin to participate. When someone 
comes forward with a brief comment after staying in the 
background for most of the study circle, you can encourage 
him or her by conveying genuine interest and asking for 
more information. And it's always helpful to talk with people 
informally before and after the session. 

Possible responses: As the facilitator, it is your responsibility 
to handle domineering participants. Once it becomes clear 
what this person is doing, you must intervene and set limits. 
Start by limiting your eye contact with the speaker. Remind 
the group that everyone is invited to participate; “Let's hear 
from some folks who haven't had a chance to speak yet” if 
necessary, you can speak to the person by name. “Charlie, 
we've heard from you; now let's hear what Barbara has to say.” 
Be careful to manage your comments and tone of voice - you 
are trying to make a point without offending the speaker. 

Possible responses: Responding to this takes judgment and 
intuition, it is the facilitator's role to help move the discussion 
along. But it is not always clear which way it is going. Keep an 
eye on the participants to see how engaged they are, and if 
you are in doubt, check it out with the group. “We're a little 
off the topic right now. Would you like to stay with this, or 
move on to the next question?" if a participant goes into a 
lengthy digression, you may have to say: “We are wandering 
off the subject, and I'd like to give others a chance to speak.” 



©Topsfield Foundation. Reprinted with permission from The Guide for Training Study Circle Facilitators 
(1998) by the Study Circle Resource Center. 



Problem: 

Lack of focus, not 
moving forward, 
participants wander 
off the topic. 



Problem: 

An aggressive or 
talkative person 
dominates the 
discussion. 
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|A| Su?9**tionj for dealing with typical challenges 



Problem: 

Someone puts forth 
information which 
you know to be false. 
Or, participants get 
hung up in a dispute 
about facts but no one 
present knows the 
answer. 



Possible responses: Ask, "Has anyone heard of conflicting 
information?" if no one offers a correction, offer one yourself. 
And if no one knows the facts, and the point is not essential, 
put it aside and move on. if the point is central to the 
discussion, encourage members to look up the information 
before the next meeting. Remind the group that experts 
often disagree. 



Problem: 

Lack of Interest, 
no excitement, no 
one wants to talk, 
only a few people 
participating. 



Possible responses: This rarely happens in study circles, but it 
may occur if the facilitator talks too much or does not give 
participants enough time to respond to questions. People need 
time to think, reflect, and get ready to speak up. it may help to 
pose a question and go around the circle until everyone has a 
chance to respond. Occasionally, you might have a lack of 
excitement in the discussion because the group seems to be in 
agreement and isn't coming to grips with the tensions inherent 
in the issue, in this case, the leaders job is to try to bring other 
views into the discussion, especially if no one in the group holds 
them. "Do you know people who hold other views? What would 
they say about our conversation?” 



Problem: 

Tension or open 
conflict in the group. 
Perhaps two 
participants lock 
horns and argue. Or, 
one participant gets 
angry and confronts 
another. 



Possible responses: if there is tension, address it directly. 
Remind participants that disagreement and conflict of ideas is 
what a study circle is all about. Explain that, for conflict to be 
productive, it must be focused on the issue: it is acceptable 
to challenge someone's ideas, but personal attacks are not 
acceptable. You must interrupt personal attacks, name-calling, 
or put-downs as soon as they occur. You will be better able to 
do so if you have established ground rules that disallow such 
behaviors and that encourage tolerance for ail views. Don't 
hesitate to appeal to the group for help: if group members 
bought into the ground rules, they will support you. As a last 
resort, consider taking a break to change the energy in the 
room. You can take the opportunity to talk one-on-one with 
the participants in question. 
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Good study circle facilitators 



• are neutral, the facilitator's opinions are not part of 
the discussion 

• help the group set its ground rules, and keep to them 

• help group members grapple with the content by asking 
probing questions 

• help group members identify areas of agreement and 
disagreement 

• bring in points of view that haven't been talked about 

• create opportunities for everyone to participate 

• focus and help to clarify the discussion, 

• summarize key points in the discussion, or ask others 
to do so 

And 

• are $e*f -aware good facilitators know their own strengths, 
weaknesses, 'hooks,* biases, and values 

• are able to put the group first 

• have a passion for group process with its never-ending variety 

• appreciate all kinds of people 

• are committed to democratic principles 



©Topsfield Foundation. Reprinted with permission from The Guide for Training Study Circle Facilitators 
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V Tips for effective 
discussion facilitation 

Be prepared. 

The facilitator does not need to be an expert on the topic being discussed, but should 
be the best prepared for the discussion This means understanding the subject, being 
familiar with the discussion materials, thinking ahead of time about the directions in which 
the discussion might go, and preparing questions to help further the discussion. 

Set a relaxed and open tone. 

• Welcome everyone and create a friendly and relaxed atmosphere. 

• Well-placed humor is always welcome, and helps to build the group's connections 

Establish clear ground rules. 

At the beginning of the study circle, help the group establish its own ground rules by 
asking the participants to suggest ways for the group to behave. Here are some ground 
rules that are tried and true: 

• Everyone gets a fair hearing. 

• Seek first to understand, then to be understood. 

• One person speaks at a time 

• Share "air time." 

• Conflict is not personalized. Don't label, stereotype, or call people names, 

• Speak for yourself, not for others. 

• What is said in this group stays here, unless everyone agrees to change that 

Monitor and assist the group process. 

• Keep track of how the group members are participating - who has spoken, who 
hasn't spoken, and whose points haven't been heard. 

• Consider splitting up into smaller groups to examine a variety of viewpoints or 
to give people a chance to talk more easily about their personal connection to 
the issue. 

• when deciding whether to intervene, lean toward non-intervention, 

• Don't talk after each comment or answer every question; allow participants to 
respond directly to each other 

• Allow time for pauses and silence. People need time to reflect and respond. 

• Don't let anyone dominate; try to involve everyone. 

• Remember: a study circle is not a debate, but a group dialogue If participants forget 
this, don't hesitate to ask the group to help re-establish the ground rules. 



©Topsfield Foundation. Reprinted with permission from The Guide for Training Study Circle Facilitators 
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Tips for effective discussion facilitation (continued) 



Help the group grapple with the content. 

• Make sure the group considers a wide range of views. Ask the group to think 
about the advantages and disadvantages of different ways of looking at an 
issue or solving a problem. 

• Ask participants to think about the concerns and values that underlie their 
beliefs and the opinions of others. 

• Help the discussion along by clarifying, paraphrasing, and summarizing the 
discussion. 

• Help participants to identify “common ground," but don't try to force 
consensus. 



Use probing comments and open-ended questions which don't lead to 
yes or no answers. This will result in a more productive discussion. Some useful 
questions include: 

• What seems to be the key point here? 

• What is the crux of your disagreement? 

• What would you say to support (or challenge) that point? 

• Please give an example or describe a personal experience to illustrate 
that point. 

• Could you help us understand the reasons behind your opinion? 

• What experiences or beliefs might lead a person to support that point 
of view? 

• What do you think people who hold that opinion care deeply about? 

• what would be a strong case against what you just said? 

• What do you find most persuasive about that point of view? 

• What is it about that position that you just cannot live with? 

• What have we missed that we need to talk about? 

• What information supports that point of view? 

Reserve adequate time for closing the discussion. 

• Ask the group for last comments and thoughts about the subject. 

• Thank everyone for their contributions. 

• Provide some time for the group to evaluate the study circle process. 



©Topsfield Foundation. Reprinted with permission from The Guide for Training Study Circle Facilitators 
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Accessing State Plans 

Below is listed contact information to help you access your state’s Adult Education plan. 
Some states have their state plans available online; in others, you would have to contact 
the office of the State Director. The directions under each state tells you whether the state 
plan is available online or how to contact the office of the State Director. The goals are 
usually listed either in a separate section (or Appendix) of the Plan or in introductory 
sections having to do with Mission/Vision. If no goals are evident in the state plan, contact 
your State Director for more information. The homepage of the Office of Vocational and 
Adult Education in the U.S. Department of Education http://www.ed.gov/offices/OVAE 
can provide additional resources. 

For Session 1 you will need to copy (or excerpt) the pages that have the goals or mission 
and make a copy for each participant. 



1. Alabama: 

Contact the office of the State Director for Alabama: 

Alabama Adult and Community Education Program 

P.O. Box 302101 

5343 Gordon Persons Building 

50 North Ripley Street 

Montgomery, AL 36130-21 01 

(334) 242-8185 

(800) 392-8086 

Toll Free Restrictions: AL residents only 

FAX: (334) 242-2236 

URL: http://www.alsde.edu/ 

2. Alaska: 

Contact the office of the State Director for Alaska: 
Adult Basic Education (Alaska) 

Department of Labor and Workforce Development 

P.O. Box 25509 

Juneau, AK 99802-5509 

(907) 465-8714 

FAX: (908) 465-4537 

URL: http://www.labor.state.ak.us/ 

3. Arizona: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 



4. Arkansas: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

5. California: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

6. Colorado: 

State plan available online at: 

http://www.cde.state.co.us/cdeadult/adultaeflastatep 

lan.htm 

7. Connecticut: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

8. Delaware: 

Contact the office of the State Director for Delaware: 
Adult Community Education (Delaware) 

P. O. Box 1402 
J. G. Townsend Building 
Dover, DE 1 9903 
(302) 739-3743 
FAX: (302) 739-3744 
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9. Florida: 

Contact the office of the State Director for Florida: 
Adult and Vocational Education (Florida) 

Bureau Chief, Program Implementation and 
Accountability 

State Department of Education 
Division of Workforce Development 
714 FEC Building 
Tallahassee, FL 32399-0400 
(850) 487-3140 
FAX: (850) 487-1 735 

URL: http://www.firn.edu/doe/bin00051/adult_ed.htm 

10. Georgia: 

Contact the office of the State Director for Georgia: 
Department of Technical and Adult Education 
(Georgia) 

Office of Adult Literacy 
Suite 400 

1800 Century Place, NE 
Atlanta, GA 30345-4304 
(404) 679-1635 
FAX: (404) 679-1 630 
URL: http://www.dtae.org 

11. Hawaii: 

Contact the office of the State Director for Hawaii: 
Adult Education Office (Hawaii) 

Department of Education 
Community Education Section 
634 Pensacola Street, #222 
Honolulu, HI 96814 
(808) 594-0170 
FAX: (808) 594-01 81 

12. Idaho: 

Contact the office of the State Director for Idaho: 

ABE Director/GED Administrator 

Department of Education 

PO Box 83720 

Boise, ID 87320-0027 

(208) 332-6931 

FAX: (208) 334-4664 

URL: http://www.sde.state.id.us 

13. Illinois: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 



14. Indiana: 

Contact the office of the State Director for Indiana: 
Division of Adult Education (Indiana) 

Room 229, Statehouse 
Indianapolis, IN 46204-2798 
(317) 232-0522 
FAX: (31 7) 233-0859 

15. Iowa: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

16. Kansas: 

Contact the office of the State Director for Kansas: 
Adult Education (Kansas) 

Kansas Board of Regents 

Suite 1410 

700 SW Harrison 

Topeka, KS 66603 

(785) 296-7159 

FAX: (785) 296-0983 

URL: http://www.ksbe.state.ks.us 

17. Kentucky: 

State plan available online at: 

http://www.state.ky.us/agencies/wforce/ 

dael%20state%20plan/kentucky_state_plan.htm 

18. Louisiana: 

Contact the office of the State Director for Louisiana: 

Division of Adult Education and Training (Louisiana) 

State Department of Education 

P. O. Box 94064 

Baton Rouge, LA 70804-9064 

(225) 342-3336 

FAX: (225) 219-4439 

URL: http://www.doe.state.la.us/ 

19. Maine: 

Contact the office of the State Director for Maine: 
Bureau of Applied Technology and Adult Learning 
(Maine) 

23 State House Station 
Augusta, ME 04333 
(207) 287-5854 
FAX: (207) 287-5894 
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20. Maryland: 

Contact the office of the State Director for Maryland: 
Division of Career Technology and Adult Learning 
(Maryland) 

State Department of Education 
200 West Baltimore Street 
Baltimore, MD 21201 
(410) 767-0168 
FAX: (41 0) 333-2099 

URL: http://www.research.umbc.edu/~ira/ 

21. Massachusetts: 

Contact the office of the State Director for 
Massachusetts: 

State Department of Adult Education 
(Massachusetts) 

Department of Education 

350 Main Street 

Malden, MA 02148-5023 

(781) 338-3850 

TTY: (800) 439-2370 

FAX: (781) 338-3394 

URL: http://www.doe.mass.edu/acls/ 

22. Michigan: 

State plan available online at: http://www.mdcd.org 
(select Adult Education Program topic) 

23. Minnesota: 

Contact the office of the State Director for 
Minnesota: 

Adult Basic Education (Minnesota) 

Department of Children, Families & Learning 

1500 Highway 36 West 

Roseville, MN 55113 

(651) 582-8442 

FAX: (651) 582-8496 

24. Mississippi: 

Contact the office of the State Director for 
Mississippi: 

Mississippi State Board for Community and Junior 
Colleges 

3825 Ridgewood Road 
Jackson, MS 3921 1 
(601) 432-6481 
FAX: (601) 432-6365 



25. Missouri: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

26. Montana: 

Contact the office of the State Director for Montana: 
Office of Adult Education (Montana) 

Office of the State Superintendent 

PO Box 2022501 

Helena, MT 59602-2501 

(406) 444-4396 

TTY: (406) 444-1812 

FAX: (406) 444-3924 

URL: http://www.metnet.state.mt.us 

27. Nebraska: 

Contact the office of the State Director for Nebraska: 
Adult Education (Nebraska) 

State Department of Education 
301 Centennial Mall South 
P. O. Box 94987 
Lincoln, NE 68509-4987 
(402) 471-4807 
FAX: (402) 471-812 7 

28. Nevada: 

Contact the office of the State Director for Nevada: 
Adult Education Office (Nevada) 

Nevada Department of Education 
Workforce Education Team 
700 East Fifth Street 
Carson City, NV 89701 
(775) 687-9104 
FAX: (775) 687-9114 

29. New Hampshire: 

Contact the office of the State Director for New 
Hampshire: 

Bureau of Adult Education (New Hampshire) 

State Department of Education 
1 01 Pleasant Street 
Concord, NH 03301 
(603) 271-6698 
TTY: (800) 735-2964 
FAX: (603) 271-1953 
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30. New Jersey: 

Contact the office of the State Director for New 
Jersey: 

Adult Education Office (New Jersey) 

Department of Education — Office of 
School-to-Career 

River View Executive Plaza Building 100 

Route 29, P.O. Box 500 

Trenton, NJ 08625-0500 

(609) 341 -3395 

FAX: (609) 984-0573 

31. New Mexico: 

Contact the office of the State Director for New 
Mexico: 

Adult Basic Education (New Mexico) 

Department of Education 

Education Building 

300 Don Gaspar 

Santa Fe, NM 87501 

(505) 827-6672 

FAX: (505) 827-4041 

URL: http://www.nmabe.org 

32. New York: 

Contact the office of the State Director for New York: 
Adult Education Program (New York) 

State Education Department 

Room 307 Education Building 

Albany, NY 12234 

(518) 474-8920 

FAX: (51 8) 474-2801 

URL: 

http://www.emsc.nysed.gov/workforce/work.html 

33. North Carolina: 

Contact the office of the State Director for North 
Carolina: 

Basic Skills Program (North Carolina) 

North Carolina Community College System 

Caswell Building 

5024 Mail Service Center 

Raleigh, NC 27699-5024 

(919) 733-7051 

FAX: (919) 733-0680 

34. North Dakota: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 



35. Ohio: 

Contact the office of the State Director for Ohio: 
Adult Basic and Literacy Education (Ohio) 

Ohio Department of Education 

Suite 210 

933 High Street 

Worthington, OH 43085-4046 

(614) 466-5015 

FAX: (614) 752-1640 

URL: http://www.ode.state.oh.us/ctae/able/ 

36. Oklahoma: 

Contact the office of the State Director for 
Oklahoma: 

Lifelong Learning Section (Oklahoma) 

State Department of Education 

Oliver Hodge Memorial Education Building, Room 

115 

2500 North Lincoln Boulevard 
Oklahoma City, OK 73105 
(405) 521-3321 
(800) 405-0355 
FAX: (405) 521-6205 

37. Oregon: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

38. Pennsylvania: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

39. Rhode Island: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

40. South Carolina; 

Contact the office of the State Director for South 
Carolina: 

Office of Adult and Community Education (South 
Carolina) 

Department of Education 
Rutledge Building, Room 902 
1429 Senate Street 
Columbia, SC 29201 
(803) 734-8071 
FAX: (803) 734-5685 



3.6 HOW ARE WE DOING? - NCSALL TEACHING AND TRAINING MATERIALS 




41. South Dakota: 

Contact the office of the State Director for South 
Dakota: 

Adult Education Office (South Dakota) 

Department of Education and Cultural Affairs 

700 Governors Drive 

Pierre, SD 57501-2291 

(605) 773-4716 

FAX: (605) 773-4236 

42. Tennessee 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

43. Texas: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

44. Utah: 

Contact the office of the State Director for Utah: 
Office of Adult Education (Utah) 

State Department of Education 
Room 234 

1234 South Main Street 
Salt Lake City, UT 841 01 -31 1 7 

(801) 538-7824 
(800) 451-9500 

Toll Free Restrictions: UT residents only 
FAX: (801) 578-8198 

URL: http://www.usoe.k12.ut.us/adulted/home.htm 

45. Vermont: 

Contact the office of the State Director for Vermont: 
Adult Basic Education (Vermont) 

120 State Street 
Montpelier, VT 05620-2501 

(802) 828-3134 
FAX: (802) 828-31 46 

URL: http://www.cit.state.vt.us/educ/abe/ 

46. Virginia: 

State plan available online at: 
http://www.ed.gov/offices/OVAE/stateplans.html 

47. Washington: 

Contact the office of the State Director for 
Washington: 

Office of Adult Literacy (Washington) 

State Board for Community and Technical College 

319 Seventh Avenue 

P. O. Box 42495 

Olympia, WA 98504-2495 

(360) 753-3662 

FAX: (360) 664-8808 



48. West Virginia: 

Contact the office of the State Director for West 
Virginia: 

Adult Basic Education (West Virginia) 

State Department of Education 
Building 6, Room 230 
1900 Kanawha Boulevard, East 
Charleston, WV 25305 
(304) 558-6318 
FAX: (304) 558-3946 

URL: http://www.neumedia.net/~lcabe1/index.html 

49. Wisconsin: 

Contact the office of the State Director for 
Wisconsin: 

Wisconsin Technical College System 

310 Price Place 

P. O. Box 7874 

Madison, Wl 53707-7874 

(608) 267-9684 

FAX: (608) 266-1690 

URL: http://www.board.tec.wi.us/ 

50. Wyoming: 

Contact the office of the State Director for Wyoming: 
Adult Basic Education Office (Wyoming) 

Wyoming Community College Commission 

Eighth Floor 

2020 Carey Avenue 

Cheyenne, WY 82002 

(307) 777-3545 

FAX: (307) 777-6567 

URL:http://commission.wcc.edu/WCCC/ABE/abe.htm 

District of Columbia 

Contact the office of the State Director for District of 
Columbia: 

Adult Education Office (District of Columbia) 
University of the District of Columbia 
4200 Connecticut Avenue, NW 
Washington, DC 20008 
(202) 274-6649 
FAX: (202) 274-6654 
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MCCALL 



National Center for the Study of Adult Learning and Literacy 

NCSALL at World Education 
44 Farnsworth Street 
Boston, MA 02210 
(617) 482-9485 
http://ncsall.gse.harvard.edu 



National Educational Research Centers 



The National Center for the Study of Adult Learning and Literacy (NCSALL) is a collaborative effort between the Harvard Graduate School of Education and World 
Education. The University of Tennessee, Portland State University, and Rutgers University are NCSALL's partners. NCSALL is funded by the Educational Research and Development 
Centers Program, Award Number R309B60002, as administered by the Office of Educational Research and Improvement / National Institute on Postsecondary Education, Libraries, 
and Lifelong Learning, U.S. Department of Education 




